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Abstract 

Inadequately managed technical risks have resulted in 
setbacks, failures and operational disasters in Department 
of Defense programs. Therefore, the purpose of this 
jpese-SHrctf was to synthesize a model that epitomizes a 
strategy for the management of technical risk. The model 
was synthesized using the three-pronged effort of: (1) a 
literature search and review to determine what previous work 
was done in risk assessment and risk management, (2) case 
studies of historical, contemporary and prospective new 
technology development programs, and (3) interviews 
predominately with Chief Scientists at the Wright Research 
and Development Center :(URDG-fr at Wright-Patterson Air Force 
Base. The model validation was via reviews of the model 
that were made by the stated interviewees. If the model is 
used as a technical management guide and decision aid by 
individual program or project managers at all levels, 
collective marked improvement in the technical risk 
management throughout the Department of Defense may be 






A MODEL FOR THE MANAGEMENT OF TECHNICAL RISK IN 


NEW TECHNOLOGY DEVELOPMENT PROGRAMS 


I. In troduction 


Overview 

Presently, a large number of military and civilian 
development programs involve new technology. However, the 
newer the technology that is being introduced into a 
development project, the greater the risks that may be 
involved with successfully managing the development 
program. Future military weapons systems may be even more 
complex. According to General William Thurman, Commandant 
cf the Defense Military Systems Management College, "These 
[future weapon] systems will create management problems 
that far surpass our contemporary management theory and 
practice" (143:12). This chapter introduces the concept of 
technical risk and its implications in the development of 
new technology. The specific problem for this research 
effort, relevant assumptions, and pertinent definitions are 
also presented. 

Background 

At any moment, national defense concerns, or suspected 


technological breakthroughs by potential adversaries, can 
mandate that development programs involving new 






technologies be undertaken by the United States. For 
instance, the Manhattan Project was undertaken by the 
United States during World War II to build an atomic bomb. 
Albert Einstein's 1939 letter to President Franklin D. 
Roosevelt (Appendix A) expressed concern over potential 
German strides toward the first atomic weapon (41:397). 
Similarly, after the 4 October 1957 launch of the Sputnik I 
satellite into Earth orbit by the Soviet Union, the United 
States reacted on 31 January 1958, by launching Explorer I 
(11:32; 42:91; 145:164). Since new technology is by 
definition largely unprecedented, new technology programs 
would incur much more technical risk than even state-of- 
the-art technology programs. At present, a host of 
relatively immature, but promising, new technologies and 
technological opportunities such as, X-ray laser 
technologies, and optical and neural computers are evident. 
This suggests that technological breakthroughs by any 
potential adversary may plunge the United States into a 
host of new technology weapon system development programs 
involving unprecedented technical risks (2:88; 21:76). For 
example, recent Soviet test flights of an experimental 
aircraft that uses hydrogen fuel, could eventually dictate 
that the United States undertake an urgent development 
project in that area. Also, if the Soviets are able to 
build and deploy an X-ray laser, then "the survivability of 
American space assets could be questioned" (69:1). 
Similarly, any advances in neural computers by the Soviets 
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cculd theoret ^cally provide them wi th crucial image and 
pattern (i.e., target) recognition breakthroughs. Further, 
with the recent development of the optical equivalent to 
the transistor, an optical computer is becoming 
increasingly feasible. Because an optical computer would 
use beams of light instead of the exectron currents that 
are used at present, an optical computer may be capable of 
"a trillion operations a second" (1:85) and of parallel 
processing. The military implications of a successful 
union of neural and optical computer technologies would be 
staggering (1:85,93; 2:88,94; 105:52,55; 123:40). 

However, urgent new technology development projects 
could arise from national involvement in, or observation of 
combat or terrorist incidents anywhere on the globe. For 
example, the development of more advanced Remotely Piloted 
Vehicles (RPV) for surveillance and penetration, may 
eventually be mandated by the effectiveness with which the 
United States and Israel employed RPVs in the Vietnam War 
and the 1973 Yom Kippur War, respectively (53:2-18,22,23; 
115:89). Also, the rate at which the "Egyptians and 
Israelis shot down their own aircraft in the 1973 Middle 
East War" (50:2) and the rate at which the Israelis shot 
down their own helicopters during the 1982 conflict with 
Syria, indicated that the technology to identify friend 
from foe (IFF) is a critical combat requirement. Those 
observations have prompted the United States to replace the 
ailing 1950's technology IFF systems with IFF systems that 
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will be appropriate to the potential 1990's combat 
environment (50:1-4). 

It is clear then that technology breakthroughs by 
potential United States adversaries in a number of emerging 
technologies, or international incidents that threaten 
national security, could necessitate urgent new-technology 
weapon development programs involving enormous technical 
risk. Many of these technologies, particularly "directed 
energy weapons . . . may revolutionize military strategy, 
tactics and doctrine" (58:120). It is therefore essential 
that an effective strategy for systematically reducing the 
technical risk inherent to new technology development 
projects be available. However significant technical risk 
is also incurred in performing major modifications to 
systems. A United States Air Force study concluded that 
"many modifications are generally as difficult as the 
original design" (111:116). 

Justification 

Department of Defense Directive 5000.1, regarding 
major and non-major weapons acquisition programs, 
acknowledges that program decisions must be made 
commensurate with technological risk (31:para 3, para 9a, 
para 9b), yet it provides no explicit process for 
systematically minimizing that technical risk. Even 
Military Standard 499A, Engineering Management , although 
stressing that technical risks must be assessed and 
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minimized, provides no explicit strategy for achieving 
technical risk reduction. 

Although a number of studies regarding technical risks 
in military programs have been done (10; 132; 144), many 
examine technical risk mainly as a contributor to budget 
overruns. While a recent defense document (36) provides 
excellent templates for the management of technical risk in 
transitioning from development to production, this is only 
a fraction of the regime wherein technical risk is evident. 
Still fewer of these documents examine technical risk from 
a historical perspective for lessons learned in minimizing 
technical risk when the technology in question is 
relatively unprecedented. 

What appears to be needed is an all-encompassing, high 
lev, technical risk, strategy that includes the extreme 
case where the development project involves totally new 
technology. By addressing this extreme case, the strategy 
should bound the less extreme cases of development programs 
that are extensions of more established technologies. 

While a number of military documents (25; 27; 39) 
provide technical checklists for technical risk reduction, 
they do not outline an explicit management process—a task 
sequence and priority—to achieve such risk reduction. As 
a resul 4 , the manager is usually left on his or her own 
when it comes to implementing a technical management 
strategy and integrating existing technical checklists into 
that strategy. With no explicit technical management 
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strategy, each technical manager must "re-invent the wheel" 
when it comes to technical risk management. 

According to Koontz, strategies are so important to 
success that 

. . . the development and communication strategies are 
among the most important activities of top managers . 

. . (the study has shown that most business failures 
are due] to lack of strategy, or the wrong strategy . 

. . without an appropriate strategy, appropriately 
implemented, failure is a matter of time. (98:281) 

Accordingly, a top level management strategy for technical 
risk reduction, particularly for projects involving 
unprecedented technology, could be of immense value to 
Department of Defense weapons acquisition. 

Statement .,of the .P roblem 

At present, it is not well known how to assess and 
manage the technical risks associated with weapon systems 
development programs which involve state-of-the-art 
technology. The technical risks are even greater when the 
development involves technology that is unprecedented. The 
question for this research was: How can the technical 
risks of development projects involving state-of-the-art or 
unprecedented technologies be minimized? Once this was 
answered, the researcher developed a strategy for the 
effective reduction of the technical risk associated with 
new-technology development projects. A model was 
synthesized to be used as an effective management strategy 
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and decision-asking aid for minimizing technical risk in 
development programs that involve new technology. 

Investigative Questions 

The research focused on the following investigative 
questions to answer the stated research question. 

1. How is technical risk defined? 

2. How is technical risk identified? 

3. What are the important factors for the assessment 
of technical risk? 

4. What are the major pitfalls to avoid in assessing 
or managing technical risk? 

5. What criteria are used to determine whether new 
technology is sufficiently established for incorporation 
into a development project? 

6. How does the Air Force Wright Research and 
Development Center Laboratories (WRDC) minimize the 
technical risks associated with new technology? 

Scope 

The risks in a Department of Defense development 
program are usually divided into cost, schedule and 
performance (i.e., technical) risks. This thesis will be 
concerned only with assessing and minimizing the technical 
risks connected with the development or incorporation of 
laboratory-demonstrated technology. Consequently, this 
thesis will not address the technical risks of achieving 
basic research objectives. Also, this thesis will not 
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examine the impact of unstable funding or wavering national 
or political commitment to an ongoing technical development 
effort. 

Limitations 

In keeping with the stated scope and assumptions of 
this research, a number of factors that have some impact on 
technical risk have not been addressed. Consequently, the 
model did not specifically include these factors. 
Nevertheless, such factors are important. Two such factors 
are national commitment and priority. Selected specific 
examples illustrate the importance of these factors. In 
September 1955 President Eisenhower assigned "the highest 
national priority" (42:79) to the research and development 
of the Intercontinental Ballistic Missile (ICBM). 

Similarly, or. 25 May 1961, President John F. Kennedy, in 
announcing the goal of a man on the moon, stated that the 
decision requires "a national commitment of scientific and 
technical manpower, materials and facilities" (139:29) . 
Clearly, the national priority and commitment to a 
particular technical endeavor will often be a determinant 
of the resources that are dedicated to. This will in turn 
hamper or help the risk management effort. Similarly, this 
research, while acknowledging the eventual effect of 
schedule and funding limitations on the technical risk of a 
system under development, did not explore such effects. 





The interviews that were done as part of the effort to 
synthesize the technical risk management model were 
predominately with Chief Scientists of WRDC Laboratoies. 
Although Mr. Cannon of the ASD Office of Development 
Planning was interviewed, no other cognizant personnel 
associated with a Systems Programs Office were interviewed. 
Therefore the interviews reflect technical risk from a 
largely laboratory point of view. 

Assumptions 

A major assumption of this thesis is that the cost and 
schedule risk of a development project or program are ". . . 
primarily attributable to the occurrence of unforeseen 
problems that are usually technical in nature" (99:223). A 
1982 Defense Science Board Task Force determined that the 
"causes of acquisition risk are technical, not managerial" 
(39:Sec 1,3) since an industrial process, composed of 
engineering and manufacturing, is involved. 

Therefore, a fundamental assumption of this thesis is 
that if the technical risk in the program is minimized, 
then the related cost and schedule risk of the program will 
be likewise minimized. Another assumption is that new 
technology being introduced into a development program has 
been proven feasible via laboratory experiments, although 
techniques for adapting or tailoring the technology to a 
specific application may not have been investigated in 
depth. This assumption is important because this thesis 

9 




primarily examines the technical risk of the development 
of new technology and not the risks of attaining basic 
research objectives. 

Further, it is assumed that the user's requirements 
for the system under development have been clearly and 
completely communicated to the agency charged with 
developing the equipment or weapon system in question. 

Last, but not least, it is assumed that new technology 
approaches cannot be avoided due to certain specified high 
performance requirements, or based on defense intelligence 
threat information. 

Acronym s 

The acronyms listed below apply to this thesis. Other 

acronyms are identified in the text as they occur. 

ASD (Air Force) Aeronautical Systems Division at 

Wright-Patterson Air Force Base, Ohio 

DARPA Defense Advanced Research Projects Agency 

DOD Department of Defense 

DOE Department of Energy 

GAO General Accounting Office 

NACA National Advisory Committee for Aeronautics 

(predecessor of NASA) 

MIT Massachusetts Institute of Technology 

NASA National Aeronautics and Space Agency 

NATO North Atlantic Treaty Organization 

NORAD North American Air Defense Command 

SDI Strategic Defense Initiative 
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SPO 


System Program Office 

SDIO Strategic Defense Initiative Organization 

WRDC Air Force Wright Research and Development 

Center (previously Air Force Wright 
Aeronautical Laboratories) 

Definitions 

For the purpose of this thesis, the following 
definitions apply: 

1. Modularity—the design practice of grouping 
hardware or software elements that perform the same 
particular function into "a self-contained unit" (29:44). 

2. Monte Carlo Method—"Any procedure that involves 
statistical sampling techniques in obtaining a 
probabilistic approximation to a mathematical or physical 
problem" (7:69). 

3. New Technology—Any significant technical advance, 
possibly stemming from a scientific breakthrough, that 
deviates significantly from the established approaches in 
the field or which represents a marked improvement over the 
current technical approach or application. 

4. Performance Risk—Synonymous with technical risk. 

5. Risk—"The probability and consequence of not 
achieving some defined program goal (such as cost, 
schedule, or performance)" (24:11-3). 

6. Risk Assessment—"The process of examining a 
situation and identifying the areas of potential risk" 

(30:Sec 15). 
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7. Risk Analysis—A detailed, systematic examination 
of the source of risks, to determine the probability and 
consequences of adverse events, in order to evaluate or 
develop alternatives (30:Sec XV,1). 

8. State-of-the-Art—Technology that represents the 
highest current knowledge in a particular technical 
endeavor. For all practical purposes, this term is 
synonymous with the term "new technology". 

9. Technical Risk—The probability and effect of not 
attaining a technical (i.e., performance) objective. The 
term includes the risk of fatal accident or injury to 
personnel as a result of system malfunction or failure. 

10. Technological Risk—Synonymous with technical 

risk. 

11. Work Breakdown Structure—A project development 
planning, control, and monitoring tool that breaks down a 
system into its component subsystems, functions (e.g., 
software, hardware) and indicates the interrelationship 
among these components (118:23). 

Summary 

In many cases high technical development risk stems 
from the newness of the technology or from the particular 
implementation despite the fact that the feasibility of 
applying underlying scientific principles may have been 
successfully demonstrated by laboratory experiments. 
Chapter I, introduced the concept of technical risk, 
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defined the problem, and specified the objectives of the 
research. Chapter II provides the methodology whereby a 
model for technical risk reduction was synthesized. 

Chapter III contains the literature search and review 
regarding risk in general, with the emphasis on technical 
risk. Chapter III also contains pertinent case studies to 
this research. Chapter IV provides the analysis 
considerations for synthesis of the model and then presents 
the model. Chapter V is the research conclusion with 
recommendations for further research. 
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II. Methodology 


Introduction 

The purpose of this thesis is to synthesize a model 
for managing technical risks associated with new technology 
development programs. To develop such a model, a three¬ 
pronged research effort was undertaken consisting of: (1) 
a literature search and review to determine what previous 
research was done; (2) case studies of selected new 
technology development projects in order to glean lessons- 
learned; and (3) model reviews and interviews with Air 
Force Wright Research Development Center (WRDC) scientists 
to refine and validate the model. Figure 1 summarizes the 
three-pronged approach that was taken. The inputs of the 
scientists to the model synthesis were critical since the. 
laboratories are "the state-of the art experts in many 
areas" (25:6, Sec 2) and because the model is primarily 
concerned with the development of state-of-the-art 
technology. Figure 2 indicates pertinent specific case 
studies that were selected. 

Based on the results of the literature review and the 
examination of selected case histories, a model was 
developed of the sequence of steps that should be followed 
in order to systematically reduce the technical risks 
involved with the use of new technology in development 
projects. The primary model objective was to provide a 
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Figure 1. Synthesis of Technical Risk Management Model 







useful strategy and process for reducing technical risk. 
That is, the emphasis in the model was on the macro-level 
of technical management strategy rather than on the more 
micro-level of technical principles and concepts, at the 
electronic or mechanical component level. 

Literature Search 

As a first step in the research, a literature search 
was conducted via the Defense Technical Information Center 
(DTIC) to determine what previous work was done in the area 
of risk assessment and management. However, the search was 
supplemented by manual approaches as necessary. The 
literature review initially covered the established 
techniques for assessing risk in general. After the 
general risk management techniques were examined, the focus 
shifted specifically to the management of technical risk. 
The emphasis here was on those methods that are used to 
minimize the risks involved with the development of new 
technology systems. 

Case Studies of New Technology Development Projects 

This second part of the thesis res arch to synthesize 
a risk management model examined significant major 
historical new technology development projects. As Figure 
2 illustrates, the cases were generally grouped under one 
of three categories: historical, contemporary and 
prospective. During the examination of each of these 
selected cases, the focus was on the technical risk 
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reduction methods that were required in order to achieve 
the development program in question. The examination of 
the selected cases was important in order to highlight 
those unanticipated pitfalls—and the method of resolution 
—which can be encountered in new technology development 
projects. Ultimately, the case studies lead to the 
formulation of technical management guidelines which 
constituted a fundamental strategy for the management of 
technical risk. This resulted, in turn, in the synthesis 
of a model which epitomized that strategy. 

Case Study Selection Criteria . The criteria for 
selecting the cases to be examined were as follows: 

1. The output of the development project must have 
revolutionized military strategy or tactics or had 
significant international (i.e., technological) 
ramifications. 

2. The technical principles (i.e., feasibility) of 
the development project in question must have been 
demonstrated prior to the start of the the development 
effort. However, the actual development of the technology 
must have involved significant technical risk. 

3. Ongoing major defense development projects may be 
selected if sufficient information is available regarding 
some technical risks inherent to those projects. 

4. The case need only meet one of the above criteria 
to be selected. 
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For example, the airplane, the atomic bomb and the 
Intercontinental Ballistic Missile (ICBM) are developments 
that meet at least one of the stated criteria. Therefore, 
these three cases were candidate cases to be examined. 

In the examination of the selected cases, one emphasis 
was placed on determining which actions were taken by the 
technical staff in question to eliminate or minimize the 
anticipated technical risks during development. In 
addition, any unanticipated technical problems that 
surfaced during the analyses or development were given 
particular scrutiny to determine whether or not such 
problems are characteristic of new technology development 
projects, and to determine how such technical problems were 
overcome in the specific case. 

Of particular concern was whether one or more 
categories of unanticipated technical risk were common to 
two or more of the selected case histories. Because the 
emphasis is on lessons-learned, the case history 
examinations that were conducted were of sufficient detail 
to elicit the genesis, effect and implications of a 
technical risk element in question from a management 
perspective. A detailed technical exposition of a case 
study was not attempted unless such a discourse was 
immediately relevant to the synthesis of the technical risk 
management model or clearly illustrated important technical 
risk reduction factors. 
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Technical Risk Management Model 


Based on the literature review and the examination of 
selected case histories, an appropriate model was 
synthesized for the technical risk assessment and 
management of new technology development programs. 

Groundrules for the Model . The following groundrules 
governed the model and guided the model synthesis: 

1. The model shall be a general technical risk 
management model; it will be adaptable to development 
programs involving new technology, as well as to 
development programs involving older, more mature 
technology. 

2. The model will primarily emphasize the strategy 
and process for minimizing technical risk rather than the 
detailed steps for technical activities (e.g., circuit 
design). However, reference to the documentation 
containing relevant detail will be provided. 

3. The model will indicate applicable military 
documents, as appropriate, to preclude unnecessary 
duplication of information contained in such documents. 

4. The model will indicate a preferred sequence of 
management tasks and possible paths to take based on the 
result of a previous task(s) in the sequence. 

5. The model shall indicate and distinguish between 
mandatory and highly recommended technical risk management 
tasks. 
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6. The technical risk management model will indicate 
priority and precedence (e.g., prerequisites) of the 
management tasks as well as recommended simultaneous 
events. 

Some examples of formats which were considered for use 
included: networks, hierarchical block diagrams, decision 
trees, and flowcharts. However, the model format was not 
limited to these since formats were selected, combined and 
supplemented to provide appropriate model detail, clarity 
and comprehensiveness (per these stated groundrules). The 
case histories were supplemented in the text with specific 
examples of technical risk situations, and management 
techniques elicited from topics other than the selected 
case histories. 

Interviews of Wright Research and Development Center (WRDC) 
Scientists 

The third part of the effort to synthesize a technical 
risk management model, consisted of structured interviews 
with the chief scientist at each of the five respective 
laboratory divisions of the Air Force Wright Research 
Development Center (WRDC) at Wright Patterson Air Force 
Base. The interviews were face—to—face and were conducted 
only after a preliminary model had been synthesized based 
on the literature review and case studies. 

The primary WRDC candidates for the interview was the 
Chief Scientist in each of the major divisions of AFWAL. 

If the Chief Scientist was unavailable for the interviews, 
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then the highest ranking individual, civilian or military 
having at least a master's degree in a technical discipline 
(e.g., engineering, science, mathematics) was to be 
selected for the interview. 

Interview Objectives . The primary objective of the 
interviews was to have the interviewee review and comment 
on the preliminary technical risk management model. All 
interviewees were given a minimum of six working days to 
review the model. A secondary objective of the interviews 
was to determine what techniques these scientists consider 
crucial for technical risk reduction in new-technology 
development projects, and to determine what techniques WRDC 
uses for managing technical risk. Comments and suggestions 
of the experts were incorporated into a revised model as 
necessary. 

Interview Questions . For the structured part of the 
interview, a list of questions were developed which were 
asked of each chief scientist. The questions asked 
attempted to elicit the following information: 

1. Technical and managerial experience of the person 
being interviewed. 

2. Examples of technical risk arising from new 
technology. 

3. The approach that is used by their laboratory 
and/or organization to identify technical risk. 

4. The approach that is used by their laboratory 
and/or organization to manage technical risk. This 
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included a determination of the approach used, if any to 
monitor the level of risk reduction. 

5. Any specific quantitative and qualitative 
approaches that are being used for technical risk 
management. 

6. Key documents, references or regulations that 
he/she cites as key guidance for providing risk assessment 
and risk management procedures. 

The unstructured part of the interview consisted of 
general question to allow the interviewee to comment on 
relevant technical risk management items that may not have 
teen touched on during the structured interview. The 
specific questions that were asked on the interview are in 
Appendix B. 

WRD C _ Revi . ew of Preliminary Model . 

After the interview, the interviewees were asked to 
provide their comments on the preliminary technical risk 
management model. In particular, the interviewees were 
asked to comment on the adequacy, utility, limitations of 
the model and to recommend specific ways that the mode^ 
could be improved. The interviewees were encouraged to 
indicate the recommended revisions by marking the model. 
The model revision was then reviewed a second time by the 
experts to validate the final product. 
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WRB C Review of Revis e d Mo del 


The model was revised based on the scientists' review, 
comments and recommendations. To verify that the 
interviewees concurred with the revised model, a follow-up 
visit was conducted to provide the interviewees with one 
final chance to comment. In the event that che individual 
who provided a comment was unavailable for the follow-up 
visit a copy of the revised model was left for review. 

In the event that any conflicting comments or 
recommendations pertaining to the model could not be 
resolved, the reason for the conflict was so noted and 
indicated on the model (e.g., by dashed lines). 

Model Review and Interviews 

An attempt was made to in\erview each of the Chief 
Scientists of the Wright Research Development Center 
Laboratories. Each interviewee also reviewed the 
preliminary model. The interviewees/model reviewers with 
their respective position titles indicated, are: 


Dr. Keith Richey Technical Director cf WRDC 

x.59400 


Dr. Harris Burte Chief Scientist, WRDC 

x.52738 Materials Laboratory 


Dr. Jim Olsen Chief Scentist, WRDC 

x.57239 Flight Dynamics Laboratory 


Dr. Arnold Mayer Chief Assistant for 

x.53311 Research and Technology, 

Vehicle Subsystems Division 
of WRDC Flight Dynamics 
Laboratory 
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Dr. Squire Brown 
x.52377 


Mr. Jack Cannon 
X. 55877 


Chief Scientist, WRDC 
Technology Division 

Technical Director of 
Development Planning, 
ASD Office of Plans 


Dr. Jess Riles 
X. 53627 


Mr. C.J. Cosenza 
x, 55393 


Chief Scientist, WRDC 
Avionics Laboratory 

Chief, Advanced Development 
Division, WRDC Technology 
Exploitation Directorate. 


The WRDC Laboratory Chief Scientists shown, reviewed 
the model and provided an interview. For ease of reviewing 
(or transcribing) the interviews, a tape recorder was used 
during the interviews with the permission of each 
interviewee. Dr. Curran, Chief Scientist of the WRDC 
Aerospace Propulsion and Power Laboratory, was unavailable 
during the period of the interviews. Dr. Olsen and Dr. 
Mayer were interviewed jointly. Mr. Cannon and Mr. 

Cosenza, were interviewed and they also reviewed the model; 
they were both recommended by one or more chief scientist. 
The phone extensions shown are those on Wright-Patterson 
Air Force Base, Ohio. 
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Background 


In 1981, US Air Force Brigadier General William 
Thurman, Commandant of the Defense Systems Military 
College, stated: 


There is an impressive and growing list of failures in 
large-scale advanced technology programs. Many of 
these failures involve military programs, but there 
have also been numerous failures in non-military 
projects. We know less than we think we do about the 
management process by which new technology is 
converted into operating systems. (143:12) 


Even though a new technology may have been 
demonstrated in a laboratory, the risks of failure in 
developing (i.e., applying) that technology may be high. 
This is because information on the use and the limitations 
of the new technology in a particular application, may be 
inadequate or unavailable. Such limited information 
imposes uncertainty, and the associated risk of failing the 
development project in question. Specific steps are 
required to systematically minimize the attendant technical 
development risks associated with new technology (80:148; 
143:12). 

The United States' Manhattan Project is perhaps the 
premier example of a new technology development program. 

The Manhattan Project was undertaken during World War II to 
design and build an atomic bomb based on the principle of 
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atomic fission that was successfully achieved in the 
laboratory. The development of an atom bomb involved 
considerable technical risk (80:180-181). A contemporary 
example of a major technical undertaking that relies 
heavily on the state-of-the-art technology, and that 
involves concomitant high technological risk is the United 
States' prospective Strategic Defense Initiative (SDI)— 
colloquially referred to as the "Star Wars" program. In 
his SDI announcement on 23 March 1983, President Ronald 
Reagan challenged the defense technological community to 
develop the technology which could "... intercept and 
destroy strategic ballistic missiles before they reached 
our own soil or that of our allies" (97:8). According to 
Dr. Donald Ulrich, senior program director for the Air 
Force Office of Scientific Research, a significant amount 
of the technical risk reduction in the SDI program depends 
on the availability of advanced materials suitable for the 
formidable space environment (84:10; 97:8,9). 

Another project which would involve considerable 
technical risk is the transatmospheric flight vehicle, 
announced in President Reagan's 1986 State of the Union 
Address. That aircraft, called the National Aero-Space 
Plane (WASP) may be able to travel from California to the 
Orient within two hours, have the capability to travel at 
"speeds (above Mach 6) in the atmosphere" (133:13), enter 
space, and then re-enter the atmosphere for a runway 
landing (140:67). 
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State-of-the-art technologies such as Very-High-opeed- 
Integrated-Circuits (VHSIC), being pursued by the Air Force 
and Navy, present significant technical risk—especially 
where semiconductor material processing and fabrication are 
concerned. Other high technology development efforts in 
the areas of high-power lasers and microwave weapon 
technology also involve high technical risk. Despite the 
hxyh technical risks, these technologies—and others—are 
being pursued because they are crucial to a strong national 
defense (16:50-53; 33:1,2; 34:1,2; 35:1,2). 

To underscore the importance of adequately identifying 
and managing technical risk, and to illustrate that 
technical risk may also be inherent to established 
technologies, one can cite two recent false alarms at the 
North American Air Defense Command (NORAD). 

On June 3, 1980, false attack indications were caused 
by a faulty component in a communications processor 
computer. On June 6, 1980, false attack indications 
were once again caused by the faulty component during 
operational testing. (61:3) 

Considering the crucial defense role of NORAD, it is 
clear that any technical performance deficiencies inherent 
to NORAD attack warning equipment, including false alarms, 
jeopardize national defense (61:3,13,22). 

Definition of Risk 

Webster defines risk as "the possibility of loss, 
injury, disadvantage or destruction" (146:1961). Still, 
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other definitions of risk focus on safety considerations. 
Risk has been defined as "The probability of loss or injury 
to people and property" (89:9). In fact, risk is often 
associated with a mathematical probability or a likelihood 
of occurrence. 

However, it is acknowledged that "there is 
considerable overlap (and often confusion) between the 
terms Reliability, Safety, Hazard and Risk" (89:9). For 
instance, the risk of fatal accident in an aircraft was 
quantified by the early 1960's as being one fatal accident 
out of every million flights (89:1). This is consistent 
with the definition that risk is "an expression of the 
possibility of a mishap in terms of hazard severity and 
hazard probability" (39:3). In the Department of Defense 
(DOD) risk has been defined as "a potential occurrence that, 
would be detrimental to plans or programs" (30:1, Sec 15). 

Haimes distinguishes between risky and uncertain 
situations as being, respectively, those in which "the 
potential outcomes can be described by reasonably well 
known probability distributions" (85:217), and those in 
which they cannot. Nevertheless Haimes acknowledges that 
the term "risk" can be used to "connote situations of both 
risk and uncertainty" (85:217). However, Golden and Martin 
assume that "uncertainty and risk can be treated as 
synonymous terms" (80:148). However, they have divided 
risk into the four categories of "environmental, 
functional, informational and technical risk" (80:148). 
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These four categories they have subdivided even further to 
create a hierarchy of risk categorization. A Rand study 
proposes a similarly structured "hierarchy of uncertainty" 
(144:24) with a probability distribution associated wjth 
the success of each component of each level of the system 
hierarchy in question. According to the Rand study, the 
collective probability of system success can then be 
obtained from these individual system components by using 
the techniques of Monte Carlo Simulation and Propagation of 
Error (144:24). 

Similarly, in the Department of Defense, weapon system 
development risk has typically been divided into 
performance, cost and schedule risks, although these risks 
are actually interrelated (28:27; 151:97). A Defense 
Systems Management College handbook on risk assessment 
techniques defines risk as the "probability and consequence 
of not achieving some defined program goal (such as cost, 
schedule, or technical performance)" (24:3, Sec II). In 
summary, most definitions of risk are centered around the 
concept of uncertainty since in the presence of certainty, 
there is no risk (80:148; 143:38). 

Technical Risks 

Technical risks are "those problems or uncertainties 
that may hinder the achievement of design and development 
goals" (58:1) of a weapon system. The technical risk in 
complex weapon system development stems from unknowns 
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connected with "... inadequate knowledge of the basic 
technology or its specific implementation" (143:12). 

The unknowns in a project nay be subdivided into 
recognized unknowns and unrecognized unknowns. A 
recognized unknown is an item for which an individual is 
aware that the present information or knowledge is absent 
or inadequate. However, an unrecognized unknown is an item 
of which an individual is not even aware is, or will be, a 
pertinent factor for technical risk reduction for the 
project or endeavor in question. Therefore the planning to 
effectively handle unknowns—information risk—is important 
for systematic risk reduction (132:38). Indeed, the 
performance risk associated with new technology stems from 
uncertainty. Rowe cautions that in addition to unknowns 
connected with pushing the state-of-the-art, technological 
risk is also due to ". . .a major gap between an 
organizations' area of expertise and what is required to 
perform effectively" (132:40). In fact, Shannon states 
that actual project failures have usually been traced to 
either 


1. Failure to seek the help of appropriate 
specialists. 

2. Failure to ask the specialists the right questions 
at the right times. 

3. Failure to heed the advice of the specialist after 
it is given. (137:83) 
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Any plans therefore should specifically address the 
availability, access to, and utilization of specified 
technical experts to minimize technical risk (79:321). 

Risk Assessment 

Risk assessment is "the process of examining a 
situation and identifying the areas of potential risk" 
(30:Sec 15,1). The risk assessment is usually followed by 
a risk analysis wherein the probabilities and consequences 
of particular events are examined and quantified. In the 
1940's technical risk management focused on the improving 
reliability through the enhancement of quality with better 
design and materials (89:2). In 1954, Air Force Brigadier 
General Bernard A. Schriever was given authority to develop 
"the free world's first ICBM [Intercontinental Ballistic 
Missile], an immensely complex technical task" (9:201). 
However, ICBM accidents drove the formalization of safety 
studies as part of system design: 

The emergence of a Systems Safety Study as an 
independent separate activity was first mandated by 
the Air Force in 1962 following disastrous accidents 
at four ICBM missile/silo complexes. (89:3) 

Ultimately, this led in 1969 to MIL-STD-B82, System 
Safety Programs for Systems and Equipment: Requirements 
for , as a documented safety approach (81:75; 89:4). In the 
1960's, due to the development of the Intercontinental 
Ballistic Missiles and manned rocket programs such as 
Project Mercury and Project Gemini, there was enormous 
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emphasis on technical performance and on the reduction of 
technical risk. This was especially the case because such 
rocket launches could not be redone once the rocket left 
the launch pad. It was in 1961 that H. A. Watson of Bell 
Telephone Laboratories originated the Missile launch 
control system. Fault Tree Analysis remains an important 
and widely used technique for technical risk assessment 
(89:3,45). General William Thurman stated that 


Formal recognition of risk and uncertainty analysis in 
DOD resulted from a 31 July 1969 memorandum from David 
Packard to the DEP SEC DEF [Deputy Secretary of 
Defense) identifying problem areas in the weapon 
system acquisition process. (143:12) 


The WASH 1400. A Reactor Safety Study , which was 
commissioned by the United States Atomic Energy Commission 
and completed in 1974, was a comprehensive risk assessment 
of nuclear facilities. That study, led by Professor N. 
Rasmussen, remains a landmark in risk assessment and risk 
analysis methodology. Many of the qualitative and 
quantitative risk assessment techniques used in the WASH 
1400 study are widely used today (89:4,19-24). The United 
States Air Force Systems Command specifies that the general 
steps for a technical risk analysis are as follows: 


1. Form a Risk Analysis Task Force 

2. Identify the objective 

3. Specify critical events 

4. Develop contingencies for each event 

5. Construct program network 

6. Collect data 

7. Evaluate network 

8. State conclusions. (25:Sec 8,7) 
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Henley and Kumamoto outlined a three phase methodology 
for conducting a technical risk analysis as outlined in 
Figure 3. Similarly, Haimes has defined risk assessment as 
a process that includes the five steps of "risk 
identification, risk quantification, risk evaluation, risk 
acceptance and risk aversion, and risk management" 

(85:217). 

Qualitative Techniques of Risk Assessment 

Qualitative risk assessment techniques are those 
techniques which rely heavily on opinions and judgements of 
experts rather than on explicit empirical data. Therefore, 
the experts must have detailed technical knowledge of the 
purpose and function of the system in question and of the 
environment in which the system must operate (141:77). 

Some clear indications of technical expertise are the 
holding of positions in national scientific organizations, 
on editorial boards for key technical journals, and the 
holding of research contracts awarded by the government 
(88:3-5). The detailed knowledge of experts is especially 
important because it is often necessary for trade-off 
decisions to be made "among conflicting . . . objectives 
and attributes" (85:217). In particular, appropriate and 
early laboratory involvement is crucial for accurate 
assessment of technical risk, since the laboratories are 
"the state-of-the-art experts in many areas" (25:6, Sec 2). 
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PHASE I: 


PHASE II: 


PHASE III 


Figure 


DEFINE THE SYSTEM 

Step 1. IDENTIFY THE HAZARD (1$ it toxic 
explosion, a fire...?) 


Step 2. IDENTIFY THE PARTS OF THE SYSTEM WHICH GIVE 
RISE TO THE HAZARDS. (Ooes it involv. the 
chemical reactor, the storage tank, the power 
plant?) 

Step 3. BOUND THE STUDY (Will it include detailed studies of 
risks from sabotage, adversary action, war, public 
utility failures, lightning, earthquakes...?) 

IDENTIFY THE ACCIDENT SEQUENCES. EVENT TREES, FAULT 
TREES 

: CONSEQUENCE ANALYSIS 


3. Risk Analysis Methodology (89:19-21,24,29) 
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Harmon states that the three characteristics of 
diversity, depth, and breadth of knowledge are important 
for "a good panel of experts" (88:4). To be diverse the 
panel should be comprised of multidisciplinary experts 
(e.g., logistics, engineering, safety, reliability, etc.) 
in order that multiple aspects of the problem or system in 
question can be completely examined. For depth, at least 
one expert should be present with profound knowledge in 
each major scientific or engineering field (29:58,59; 88:3- 
5). Finally, for breadth, there should be some "systems 
experts . . . [since they are] accustomed to thinking . . . 

in terms of the interactions of various subsystems" (88:4). 

Design Reviews . Often, experts are assembled in a 
conference or committee to collectively assess the 
technical risks of the system or event in question and to 
recommend actions to prevent or reduce technical risk by 
appropriate design. These conferences usually constitute 
one or more formal design reviews of a system. The conduct 
and content of design reviews for weapon systems has been 
formalized in the US Air Force in Technical Reviews and 
Audits for Systems, Equipments, and Computer Softwar e. MIL- 
STD-1521B (28: 1-123; 38-.A-7). The assessment of risk 
should also include dialogue with the intended users and/or 
customers of the system in question, to include their 
attendance at design reviews, since such information could 
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enhance the technical risk assessment and reduction effort 


(30:2,Sec XV; 89:19,24). 

However, the knowledge of even experts is incomplete 
therefore "technical errors and inconsistencies" (132:40) 
are still possible. Fox stresses that technical personnel 
can be overly optimistic regarding what is within the state 
of the art and that this over-optimism may indirectly lead 
to technical risk since immature technological approaches 
may be pursued (48:128). For example, a GAO evaluation 
cited that problems in the development of the Inertial 
Upper Stage used to boost satellites from low earth orbit, 
were actually due to "the main contractor's underestimating 
the technical complexity of the Inertial Upper Stage" 
(49:61). Even experts very close to a prospective 
technical project may disagree on the technical risks. For 
instance, Mr. Roy Woodruff allegedly resigned from the 
Lawrence Livermore National Laboratory late in 1985 because 
he claimed that "overly optimistic" (69:6) claims were 
being made by Dr. Edward Teller and Dr. Lowell Wood 
regarding the development potential for the X-ray laser 
(69:1-6,10; 100:52). The stated Dr. Teller is the eminent 
scientist who has been called the "father of the Hydrogen 
Bomb" (87:7; 100:52). 

Delphi Technique . For adequate risk assessment, steps 
must also be taken to minimize certain negative group 
processes such as external or internal pressure for a 
consensus among the experts, in order to ensure 
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objectivity. Accordingly, the opinions of one or more 
dissenting experts in such a conference must be given 
careful consideration. The Delphi technique, whereby 
expert inputs are obtained in writing without a conference 
among the experts, may be employed to solicit risk factors 
or assessments that are almost devoid of group pressure to 
conform to a particular viewpoints. Instead the Delphi 
technique arrives at a consensus among the experts by "an 
anonymous process" (136:643) whereby several cycles of 
written questions and written sunmary of anonymous 
responses (and the accompanying rationale) are presented to 
each of the experts. Based on the anonymous responses each 
expert is allowed to modify his/her answer in successive 
cycles. Ultimately, a consensus of the technical risks are 
achieved (136:643; 141:77). 

To conduct a qualitative risk assessment, the experts 
must define the system boundaries and broadly identify 
potential system malfunctions or catastrophic failure 
(e.g., explosion, fire, toxicity). Then, the respective 
system components whose failure can ultimately result in a 
particular hazard or malfunction, are identified. Finally, 
the scope of the study must be appropriately limited. 
Checklists providing some possible system hazards and 
malfunctions for consideration, are often used to 
facilitate the experts' qualitative risk assessment. A 
checklist used by Boeing Aircraft Company is presented in 
Figure 4. If the stated qualitative risk assessment 
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Hazardous Energy Sources 

1. Fuels 

2. Propellants 

3. Initiators 

4. Explosive charges 

5. Charged electrical 
capacitors 

6. Storage batteries 

7. Static electrical charges 

8. Pressure containers 

9. Spring-loaded devices 
<0. Suspension Systems 


Hazardous Processes and 



t. Acceleration 

2. Contamination 

3. Corrosion 

4. Chemical dissociation 

5. Electrical 
shock 
thermal 

inadvertent activation 
power source failure 
electromagnetic radiation 

6. Explosion 

7. Fire 

8. Heat and temperature 
high temperature 
low temperature 
temperature variations 

9. Leakage 


11. Gas generators 

12. Electrical generators 

13. rf energy sources 

14. Radioactive energy 
sources 

15. Falling objects 

16. Catapulted objects 

17. Heating devices 

18. Pumps, blowers, fans 

19. Rotating machinery 

20. Actuating devices 

21. Nuclear devices, etc. 


10. Moisture 

high humidity 
low humidity 

11. Oxidation 

12. Pressure 

high pressure 
low pressure 
rapid pressure 
changes 

13. Radiation 

thermal 

electromagnetic 

ionizing 

ultraviolet 

14. Chemical replacement 

15. Mechanical shock, 
etc. 


Figure 4. Checklists of Hazardous Sources (89:20) 


39 




process extensively identifies the alternative sequence of 
events that lead to a particular malfunction or 
catastrophic failure, as well as possible corrective action 
for the system in question, then the assessment is called a 
Preliminary Hazard Analysis (PHA). The PHA usually 
involves a decision tree which aids in interrelating the 
potential decisions and corrective actions to prevent a 
particular system malfunction or catastrophic failure 
(89:21) . 

Failure Modes and Effects Analysis . Another risk 
analysis technique is the Failure Mode and Effects Analysis 
(FMEA). The FMEA is a systematic examination and analysis 
of the worst-case impacts on the system due to the failure 
of each component in a system. As part of the FMEA, the 
potential failure modes of each component are considered 
and the applicable corrective action is recommended. Based 
on the results of the analysis, the components are assigned 
Criticality 1, 2 ,3 or sometimes 4 (89:31). 

Regarding the space shuttle, NASA designates a system 
component as Criticality 1 if a single failure in a 
component can result in "loss of life or the vehicle" 
(67:8). A component is designated as Criticality 2 if a 
single failure in the component could "cause loss of 
mission" (67:8). Those components remaining after the 
Criticality 1 and 2 designations have been completed are 
designated as Criticality 3. However, NASA also uses a 
category 1R which indicates that a particular item has one 
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or more backup (i.e., redundant) item and that failure of 
the primary as well as all redundant items would cause loss 
of life or vehicle. Similarly, category 2R indicates that 
the primary as well as all redundant components must fail 
in order to cause loss of mission (67:8; 89:31;). 

Fault Tree Analysis . This technique involves 
specifying an undesirable event as a top event and then 
constructing lower level failures that can lead to the top- 
level undesirable event. Henley points out that one 
limitation of fault trees is that as they get very large, 
"mistakes are difficult to find, and the logic becomes 
difficult to follow or obscured" (89:48). 

Material Risk . The degree of technical risk for a 
development project can often be assessed by determining 
whether materials having the required performance 
characteristics, and the processes for forming these 
materials are readily available and have been in relatively 
common use. If present materials are known to be marginal 
for the performance regime that is anticipated for the new 
technology system, then it is clear that there will be 
technical risk connected with the selection and processing 
of newer materials. For instance, aircraft evolution has 
been, and continues to be very dependent on the state of 
materials technology. This is because the greater the 
speed at which an aircraft flies, tne higher is the skin 
temperature of the aircraft as a result of air friction 
(140:67,68). Increases in aircraft speed are often due to 
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improved propulsion technology. Such improvements usually 
subject the engines to greater internal operating 
temperatures, thus height ending the technical risk. The 
technical risks associated with aerospace vehicles is 
further heightened because "high operating temperatures 
correspond to [greater] fuel efficiency" (19:52). Further, 
the need for high strength coupled with light weight in 
order to increase payload capacity (i.e., number of 
passengers, instrumentation or warheads), may mean that the 
use of new or exotic materials are mandatory in order to 
meet system performance requirements (19:51). 

Dr. Robert Barthelemy, manager of the National Aero- 
Space Plane (NASP), presently in the research phase, says 
that the availability of the appropriate high technology 
materials is crucial for reducing the technical risk 
connected with the NASP vehicle, significant technical 
risk arises from the NASP requirements to travel to the 
edge of space, and to travel faster than Mach 6 in the 
atmosphere (133:13). 

Technical Maturity . According to Major General Thomas 
R. Ferguson, Deputy Chief of Staff for technology and 
requirements at Air Force Systems Command, it is necessary 
to " . . .ensure that the technology is as mature as 

possible in order to reduce risk when we start a new 
program" (96:7). Indeed, the maturity of a candidate 
technology—which he dubs "technology readiness" (96:6)— 
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actually determines the viability of development 
alternatives. 

Therefore, a technical risk assessment must include an 
evaluation of whether materials with the required 
characteristics are available, have been tested, and 
whether these candidate materials have been previously 
applied with the required degrees of success (19:52,54; 
133:13; 140:68). 

Process and Productivity Risks . Technical risk arises 
also from new processes used to form materials into the 
respective system components. For instance, the now famous 
Lockheed "Skunk Works", which built such advanced aircraft 
like the Mach 3+ SR-71, initially had significant problems 
working with a particular titanium alloy. In spite of the 
fact that there were ten years of research on that titanium 
alloy, there had been no use of the alloy in a development 
project up to that point. Regarding the titanium alloy, 
Kelly Johnson, Skunk Works manager, stated that there was 
". . . a hell of a gap between che research and the 

application" (8:256). The crucial nature of new processes 
as sources of technical risk is also underscored by the 
initial difficulty of obtaining semiconductor-grade silicon 
with "parts-per-billion impurity levels" (18:9). Indeed, 
Mayo has observed that each advancement in electronics, 
communications and computers has "been associated with new 
materials or processing methods that are more sophisticated 
than the past" (107:59). The foregoing examples illustrate 
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that technical risk often depends on whether or not a 
particular weapon system or component is producible. In 
fact a recent General Accounting Office (GAO) report to the 
Congress (Reference COMP) determined that a smooth 
transition from weapon system design to manufacture is 
crucial tor low technical risk. As early as possible, the 
risks attendant to a particular design should be 
identified. This requires that one determines 

. . . how each part will be made, and what it will be 
made of, identifying necessary production equipment 
and skills, determining the layout and sizing of the 
facility, and determining how and when to inspect for 
quality. (20:2) 

The selection of a particular design is a key 
determinant of the degree of technical risk that is 
associated with production the weapon system or component 
in question. Accordingly, design reviews must also include 
evaluation of factors (20:2,3,9,21,22,27). In addition, 
frequent revisions to the design of a contemplated weapon 
system greatly increase the productivity risk because such 
changes can invalidate previous productivity planning and 
preparation. This is especially the case if the technology 
involved is pushing the state-of-the-art or if the design 
changes are occurring relatively late (20:31-33). All 
design changes therefore drive updates and reevaluation of 
previous analyses (28:66). 

Quality ♦ However, the technical risk arises not just 
from whether or not a particular design is producible, but 
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also from whether or not the item can be produced with the 
desired level of quality that is required or requested. 

For example 

. . . missiles are complex systems with numerous 

components and parts required in their assemblies. 

Each requires production processes and many thousands 
off individuals using an arr y of production and 
testing equipment, to manufacture and deliver a 
quality product. The opportunities for defects to 
occur are immense and one defect can render a missile 
ineffective. (66:4) 

Specifically, defective material and workmanship during the 
building of a system can contribute markedly to the 
technical risk. A recent General Accounting Office (GAO) 
report states that "soldering defects can and have lead to 
missile failures" (66:28). Further technical risk arises 
from the long term deleterious environmental effects on the 
solder joints. This is of special concern since some 
missiles, such as the Phoenix missile, have over 60,000 
solder joints (66:28-30). 

Prerequisite Technologies . An adequate technical risk 
assessment must also take into account whether certain 
prerequisite technologies required for the success of the 
project in question are sufficiently well developed. For 
instance, early computers were built using vacuum tubes. 
However, more significant computer advances were achieved 
only with the advent of the transistor (8:253,266,257; 

18:6). 
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Human-Induced Malfunctions 


A GAO study emphasized that human factors and 
reliability concerns are often inadequately considered 
during the design of a system. In fact, some studies have 
found that "human errors account for at least 50 percent of 
the failures of major systems" (54:27). Those errors 
represent a major source of risk to proper system 
performance. Because the human is often a component of the 
system performance, the reliability of the human should be 
considered as a component of the system technical risk. 
However a mission or system failure may be caused by a 
human performing a procedure incorrectly or selecting the 
wrong procedure although this would not necessarily lead to 
equipment failures (54:28-30). Human errors fall into the 
five categories of 

1. Failure to follow procedures 

2. Incorrect diagnosis of a particular situation 

3. Misinterpretation of communication (Written or 
verbal) 

4. Inadequate support, tools, equipment and 
environment 

5. Insufficient attention or caution. (54:30) 

For correct system design the designer must take into 
account certain key characteristics of the personnel that 
will be operating and maintaining the system. Some of 
these key human factors are 

1. Muscular strength and coordination 

2. Body dimension 

3. Perception and judgement 

4. Sensory capacities 
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5. Native skills and capacity to learn new skills 

6. Optimum workload 

7. Basic requirement for comfort, safety, and freedom 
from environmental stress. (54:29) 

Quantitative Techniques of Risk Assessment 

As stated above, the opinion of experts is often a 
critical component of a qualitative risk assessment. Such 
qualitative assessments are often a prerequisite for 
follow-on quantitative risk assessments (24:11-1; 89:24; 
132:40). The major techniques in use today for 
quantitatively assessing project risk are: 

1. network analysis, 

2. the method of moments, 

3. decision analysis, 

4. Work breakdown system simulation, 

5. Graphics, 

6. Estimating relationships 

7. Risk factors. (24:Gee IV,I) 

Although these techniques focus largely on cost and 
schedule risk, a technical assessment of some sort is 
normally a component of each of these techniques. 

Therefore, these techniques will be examined here. Since 
the techniques are discussed in some detail in Risk 
Assessment Techniqu e s; A Handbook for Program Management 
Personnel (24) only a summary description of each will be 
given here. 

Network Analysis . Network analysis gained renown 
because of its use in the development of the Polaris 
submarine. A network is composed of a series of lines and 
circles which are present, respectively, the activities and 
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the important intermediate objectives which will 
collectively result in the successful achievement of the 
project. In particular, such activity lines and objective 
circles must reflect the interrelationships and the 
sequence that is required for project success (24:IV-2-5). 
The Program Evaluation and Review Technique (PERT) is a 
network technique that was primarily used as a tool for 
managing project schedule. Time durations for completing 
each activity, and the respective probabilities for on-time 
completion of each of the activity are assigned to 
determine which objective is at risk of being achieved 
late, and to assess the schedule risk posed to subsequent 
or interrelated objectives. Computer simulation can then 
be used to evaluate the pessimistic, most likely, and 
optimistic completion times for the entire project 
(124:216-221). 

Method of Moments . The method of moments is a program 
risk assessment method which is primarily concerned with 
cost risk. This method, unlike the network method just 
described, does not use a sequence of network relations. 
Instead, the collective effect of the probability 
distribution functions associated with program elements in 
a Work Breakdown Structure, are determined mathematically 
instead of through simulation (24:IV-10,IV-11). 

Decision Analysis . Decision analysis is a risk 
assessment method whereby decisions are broken down into 
sequences of constituent and supporting decisions. Such 





decisions (i.e., outcomes) are usually represented by a 
"tree" where the branches from each node represent possible 
outcomes. Probabilities of each of the outcomes are 
usually assigned and expected values for each outcome can 
then be calculated. Specific variations and enhancements 
allow decision trees to be used for calculating and 
comparing the expected values for a number of different 
decision sequences regarding cost and schedule (24:IV-7). 

WASH 1400—the landmark risk assessment study 
mentioned previously—used variations of decision trees to 
assess the risk of accident whereby a nuclear reactor would 
release toxic radioactive fission products (89:24). 

Work Breakdown Simulation . The work breakdown 
simulation method is essentially the same as the method of 
moments except that computer simulation is used to obtain a 
probability density function (24:IV-13). 

Graphic Method . The graphic method is another method 
that is used to assess the cost risks of a particular 
program. In this technique, the cumulative distribution 
functions (CDF) of the individual cost elements (cost 
versus the probability of that particular cost) is combined 
to form a collective CDF for the overall program (24:IV- 
14,15,16,17). 

Estimating Relationship Method . The estimating 
relationship method is a technique that uses appropriate 
historical cost data to determine the amount of contingency 
funding that should be reserved to cover the unanticipated 
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development expense due to the technological risk involved. 
This technique requires that factors such as "engineering 
complexity, degree of system definition, contractor 
proficiency/experience, and multiple users" (24:19) be 
evaluated (24:18,19). 

Risk Factor Method . The ris.k factor method uses a 
technical Work Breakdown Structure that assigns relative 
weighting corresponding to the degree of anticipated 
technical risk associated with the system components. The 
weighting allows costs estimates for development costs to 
be assigned commensurate with technical risk, thereby 
minimizing the risks that the estimated costs are not 
realistic (24:IV—21). There are a number of computer 
programs presently in use in which the appropriate input 
information results in an output which identifies the cost 
risk to a project. For instance the Army has developed the 
Total Risk Assessing Cost Estimates (TRACE) software to 
identify the cost risks of projects in an entire command 
and to aid in assigning funds commensurate with the cost 
risks (24:23; 30:Sec 15,13). 

Design Comparisons . One method of assessing technical 
risk is by comparing the design of an existing or proposed 
system to the design of a system that has had a performance 
failure. Such a comparison may be quantitative qualitative 
or both. This technique has been used to determine the 
accident risk of a number of United States nuclear reactors 
by comparing them to the reactor which had an accident near 
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Chernobyl in the Soviet Union. Refer to the case study on 
the Chernobyl Nuclear Reactor Accident for an example of 
how design comparisons are made (62:1-6; 63:1-6). A 
comparison between an existing and a proposed design or 
system is also valuable for ascertaining that the time 
allotted for achieving the proposed design or system is 
realistic. An unrealistically short time could increase 
technical risk since some recommended technical tasks could 
be abridged or eliminated. For example, a GAO report cited 
that the time scheduled for the development of a 
particular trainer aircraft was "probably too short given 
the history of problems with engine development programs" 
(75:16). Because time is often a factor in technical risk 
reduction, technical productivity aids should be identified 
and used. One important class of such productivity aids 
are design tools which "cover the spectrum from computer 
simulations supporting design, to computer-integrated 
manufacturing" (29:82). These tools have been cited for 
increasing the "capability of engineers [by] 300-3500%" 
(29:84) . 

Risk Management 

In general, any action which reduces the uncertainties 
or unknowns associated with achieving a desired outcome is 
in essence a risk management technique. Accordingly, a 
risk assessment "is the first step in risk management" 
(30:Sec 1,2). A study sponsored by the Defense Advanced 
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Research Projects Agency (DARPA) defined risk management, 
as it regards a project, as a process of evaluating, 
decreasing or avoiding the uncertainties associated with 
meeting objectives so that a program will successfully 
achieve its goals (113:1). 

Techniques of 'technical Risk Management 

Idaally, the best method of managing technical risk is 
to eliminate risk by use of an appropriate design. This is 
especially preferred where safety risk (i.e., a hazard) is 
concerned. However, rarely will all risk be eliminated, 
therefore some technical risk management will have to be 
pursued (3:113; 36:6). Incidentally, it is highly 
important that the key technical criteria and parameters 
are identified for a system undergoing development that can 
be used as indicators of system technical health, progress 
or lack of progress (25:Sec 8,6). The following general 
types of technical risk reduction techniques have been 
cited: 

1. Test and Evaluation 

2. Use of Prototypes and Demonstration 

3. Studies and Analyses 

4. Development of New Ideas and concepts. (143:21) 

The Air Force MIL-STD-1521B cites the following 
technical risk reduction techniques: 

1. Adequate trade-offs (particularly for sensitive 
mission requirements versus engineering realism 
and manufacturing feasibility to satisfy the 
production capabilities); 
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2. Subsystem/component hardware; 

3. A responsive test program; 

4. Implementation of comprehensive engineering 
disciplines (e.g., worst case analysis, failure 
mode and effects analysis producibility analysis 
and standardization). (28:26) 

Risk reduction may also be achieved by selectively 
applying military specifications and standards, when 
appropriate, since they are a "valuable corporate memory 
[based on] . . . past experience"(25:Sec 8,8). 

Design . A fundamental guideline of risk management by 
design is sim lification. That is, the design of the 
system should be "no more complex than necessary to perform 
the function reliably" (29:39). By minimizing the number 
of parts and interconnections, simplifying the support 
tools and equipment, and providing easy access to test 
points, the system will be "simpler to build, operate, 
support and upgrade" (29:40). Modularity simplifies the 
system design, troubleshooting, and software and hardware 
revision. Software modularity also reduces the likelihood 
of logic errors and aids troubleshooting (29:44,45). For 
maximum benefit, analyses such as Fault Tree Analyses and 
Failure Modes and Effects Analyses, should be done early 
enough so that the results can be used "to influence the 
design, not just to document it [the design]" (29:67). 

Since analyses may indicate system deficiencies and 
inadequate design margins, the timing of analyses are very 
important for technical risk reduction (29:65-67). One 
basic risk reduction technique is to incorporate design 
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margins whereby parts are selected which are rated to 
withstand much higher loads or capacity, as applicable, 
then they are expected to encounter when used in the 
system. This practice in electronic design is called 
derating, and results in increased reliability (i.e., 
decreased failure rate) since the parts "experience stress 
levels below their rated values” (29:78-79). 

Testing . Testing has been identified as so crucial to 
the technical risk reduction process that the Department of 
Defense position is that: "testing will begin as soon as 
possible ... to reduce acquisition risk and to estimate 
the capability of the system under development”.(32:2). 

For example, testing revealed that the M-l tank met 
such combat requirements as firepower and mobility, but 
that the tank's power train did not meet the Army's 
durability goal (59:ii). Similarly, laboratory tests of 
the Army's Viper—a light anti-tank weapon fired from the 
shoulder of an infantryman—revealed that "static 
electricity or radar waves may cause the Viper to 
accidentally fire" (58:48). Operational tests revealed 
that" 20 of 368 Viper rounds failed to fire on the first 
attempt . . . (due to] poor quality . . . defective 
batteries . . . gunner error . . . [or] unknown [causes]" 
(58:49). For effective testing to be performed, detailed 
test planning must be completed so that the necessary test 
resources can be identified or developed, and the critical 
test issues can be ascertained (52:ii). 
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It is important that the testing of a system or 
component be very realistic. Equipment may function well 
at ambient temperature but may fail to function correctly 
when subjected to the extremely cold or hot temperatures 
that the equipment is likely to encounter. For example, 
the Air Launched Cruise Missiles' (ALCM) Flight Data 
Transmitters (FDT) were sensitive to cold temperatures. In 
two different operational launch attempts, an FDT 
malfunctioned during the pre-launch test due to the effects 
of cold temperature. Both FDT's appeared to function 
properly when tested on the ground at ambient temperatures, 
but, when later chilled in a laboratory, both failed 
(73:13). The sensitivity of FDT's to cold temperatures was 
especially important since "SAC bombers, which launch the 
ALCM, flying at the 32,000-52,000 feet required for 
strategic missions, could encounter temperatures as low as 
below [minus] 116 degrees F" (73:13). 

A fundamental example of the importance of test 
realism in reducing technical risk is illustrated by the 
case of the Army's High Mobility Multipurpose Wheeled 
Vehicle (HMMWV). Although the HMMWV performed well during 
development testing, several deficiencies became evident as 
a result of the more realistic operational testing. During 
the operational tests it was discovered thrt the vehicle's 
radiator "was subject to clogging in dusty and sandy 
environments" (51:5). Further, the HMMWV's air induction 
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system allowed "dirt and water to enter the engine" (51:5). 
This was a serious deficiency since the vehicle would be 
vulnerable to mud, dust and water during cross-country 
maneuvers. To resolve the stated deficiencies, and to 
thereby reduce the technical risks, corrective design 
modifications were planned for the HMMWV prototype (51:5- 
8 ) . 

The operational testing is normally conducted by an 
independent organization because one cannot "expect a man 
or organization who created a system to discover its 
faults" (125:6). However, the. performance of any test does 
not actually eliminate technical risk. Only the 
implementation of corrective actions or design 
modifications that are shown to be necessary by the 
testing, will actually reduce technical risk (82:258-268). 

Simulation and Analysis . Since a test article may not 
be available at the inception of a new technology 
development program, there should be heavy emphasis on 
simulation, modeling and analysis to begin to assess the 
technical risks and limitations of prospective design 
approaches (32:7). 

However, the deficiencies in a simulation may result 
in the introduction of additional technical risk. For 
example, the GAO found that "certain target and simulation 
shortcomings" (71:3) meant that some specific Anti- 
Submarine Warfare simulations, being performed by the Navy 
o test the Advanced Capability and MK—50 torpedoes, 
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1. Is the system operated by typical units? 

2. Operated by typical operating personnel? 

3. Supported by typical support units? 

4. Equipment put under realistic stress by design? 

(e g. outer envelope ol performance requirements tested?) 

5. Personnel put under realistic stress by design? 

6. Realistic combat tactics? 

7. Oid the physical environments approximate the intended range of 
envuonments (terrain, lime of day, weather, sea state, clutter, 
iff)? 

8. Oid target systems approximate actual targets, realistically 
employed? 

9. Oid threat systems approximate actual threat, realistically 
employed? 

10. Wat the tested system production representative and prepared for 
the test in a realistic manner? 


Figure 5. Framework for Evaluating the Effectivene 
of Operational Test and Evaluation (76:34-35) 
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incompletely represented "important environmental factors 
. . . and may increase technical risks of weapons not 

performing as intended" (71:3). Therefore, effective 
evaluations of test resources are necessary to determine 
the utility and validity of the test results that are 
obtained using a particular test resource (71:4). 

In particular, certain weapon systems must be tested 
in the electronic warfare environment since their 
performance may be markedly degraded in such an 
environment. Simulated threat environments which are 
inaccurate could introduce technical risk (52:2). For 
example. 


. . . the Air Force's Sparrow air-to-air missile was 
tested against aerial targets that did not 
realistically represent the actual threat. When first 
used in Vietnam, the Sparrow missile missed its target 
more than it hit. (52:6) 

A relatively new technical and test concern is whether or 
not the crucial control electronics of a particular system 
can continue to function when subjected to the 
electromagnetic pulse radiation from nuclear explosions in 
air or space (57:21; 110:164). 

Goodman points out that it is prudent to be aware that 
one or more technical approaches may look good on paper but 
must be tested to ascertain that these contemplated 
approaches can be reduced to practice. In some cases, he 
states, analyses are inadvisable for assessing the 
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technical risk and some sort of test must be implemented to 
gauge technical risk (38:A-28; 58:258-261). 


Detailed Failure Analyses . The availability of 
facilities to conduct failure analyses on subsystems or 
components of prototype hardware can also contribute 
immensely toward risk reduction since the lessons-learned 
can be used to improve the technical design. For instance 
the Rome Air Development Center has a Quick Reaction 
Failure Analysis Laboratory that can conduct detailed 
analysis of failures in semiconductor devices. Based on 
the results of failure analyses, appropriate design changes 
can be made by the DOD program in question. Therefore the 
reduction in technical risk that may be derived from 
labora-tory support can include the use of sophisticated 
laboratory instruments (e.g., scanning electron microscope, 
pattern generators, oscilloscopes, infrared scanners) and 
non—destructive tests (e.g., X-rays) for electronic 
microcircuit failure analysis, as well as technical 
assessments of contemplated design approaches (40:42,43). 

Failure analysis personnel may also be employed to 
prevent certain failures. For example, the growth of "tin 
whiskers" within some electronic components is an incipient 
failure mode that can ultimately result in internal 
shorting in hybrid and integrated circuits. This means 
that weapon systems that have tested functional before 
storage could potentially fail those same tests after 
storage. Fortunately, the Department of Defense has been 
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alerted of this particular failure mode and can minimize 
the risk by selecting proper materials. Refer to Appendix 
C for detailed correspondence on the "tin whisker" failure 
mode. Similarly, Appendix I outlines the risk of 
electronic component damage due to electrostatic discharge. 

Software Testing . Thorough testing of software, as 
part of software verification and validation, has been 
established in numerous technical programs as indispensable 
for precluding low software reliability and to verify that 
user requirements have been achieved by a system in 
question. A GAO report cited that the development of 
application software is such a laborious "and error prone 
process, and errors can be made both in deciding what the 
program should do, and in writing them to do it" (56:1). 

To enhance the testing objectivity and to increase the 
chances that user requirements were correctly understood 
and implemented in the software, independent testing and 
evaluation of the software should be conducted in addition 
to in-house testing (56:38,39). Due to the labor intensive 
nature of software, test tools—such as test data 
generators—should be used both to increase the software 
testing effectiveness and the productivity of the software 
effort. However, manual techniques can be used to 
supplement the automated testing techniques. Further, 
adequate personnel and time should be allotted for the 
software effort (56:10,12). 
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Contingency Planning . Another method of minimizing 
risk is to plan for contingencies. Thici involves looking 
at possible "what-if" situations and laying out possible 
courses of action before the particular situation occurs. 
This contingency planning may be taken further by 
evaluating the options corresponding to certain worst-case 
scenarios (28:24; 98:280). Goodman states that as a 
contingency measure, it is highly important to identify and 
carry along (i.e., demonstrate) at least one lower 
technology approach corresponding to each high technology 
approach in order to have a fallback position if the high 
technology approach is not achieved (79:262). 

Such contingency planning should include a 
consideration of relatively mundane sources of technical 
risk. For example, the 21 July 1961 flight of the Mercury 
capsule into space made Captain Virgil Grissom the second 
American in space. There had been concern that upon 
splashdown in the ocean, the space capsule would become 
submerged (instead of floating on the surface). To allow 
the astronaut to escape, an innovative explosive cord had 
been placed around the hatch. However upon splashdown, the 
hatch apparently blew inadvertently, causing sea water to 
enter the capsule. To avoid sinking with the capsule, 
Grissom quickly scrambled out of the open hatch. Grissom's 
was designed to allow him to float on water, by closing off 
a "neck dam" (23-.XVII). However, in quickly leaving the 
sinking capsule, Grissom forgot to close off "an intake 
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port in the center of the suit [and] . . . water began to 
enter the suit" (23:XVII). Having endured the 6 g force of 
ascent and the 10.2 g forces of re-entry, it was clear 
according to one doctor that Grissom came close to drowning 
during the splashdown (23:XVII,XVIII). 

Parallel Effort . To reduce the technical risk, 
"simultaneous parallel efforts are encouraged [early in a 
DOD Acquisition process) ... to reduce the risk from 
uncertain technology and development" (143:15). Such 
parallel design efforts pursue the same technical 
objectives in order that a number of different design 
approaches are available to be evaluated for superiority 
(143:15). 

For example, on 4 October 1957 the Soviet Union was 
the first to put a man-made satellite—Sputnik—into orbit. 
The United States' first attempt to achieve the same feat 
ended when the Vanguard rocket exploded on the launch pad 
during the attempted launch on 6 December 1957. However, 
the United States Army had been building the Explorer I 
vehicle which was successfully launched on 31 Januavy 1958. 
The nearly simultaneous availability of both the Vanguard 
and the Explorer I launch vehicles provided alternatives 
for achieving the United States launch objective 
(86:148,160,162). 

Incremental Development . The use of an incremental 
development strategy can greatly reduce technical risk. 
Under this approach only established technology having 
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little development risk would be implemented in an ongoing 
program and each successive version of the hardware could 
include newly matured, proven technologies and 
enhancements. A variation of this approach is called Pre- 
Planned Product Improvement, whereby the option to upgrade 
the completed or deployed system to higher technology would 
have been preserved all along by appropriate design and 
planning. An example of the incremental development 
approach was the United States Gemini space vehicle series. 
The Gemini space vehicle "drew heavily on proven Mercury 
[predecessor program] technology" (92:65), therefore the 
Gemini space vehicle were much more advanced and versatile 
(132:33). 

Technical Enrichment . Goodman has stated that one of 
the techniques of minimizing technological risks is to 
ensure that the technical, human resources involved with 
the project are continually being enriched by innovative 
ideas and perspectives. To achieve the said enrichment, 
new technical personnel can be recruited into the 
organization, attendance at professional technical meetings 
can be encouraged, and an appropriate emphasis on training 
can be employed. However, Goodman repeatedly emphasizes 
that for risk reduction, "the key management task is to 
create an early learning path" (82:258). Especially if the 
staff involved is new or inexperienced, Goodman stresses 
that it is necessary to "rush-to-prototype" (82:262) to 
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provide the staff with experience that is specific to the 
technical endeavor. 

Case Studies 

A number of case studies of technical risk management 
have been selected based on the criteria specified in 
Chapter II. These case studies reveal significant lessons 
and potential pitfalls in the management of technical risk. 
Since the case studies encompass past, contemporary and 
prospective systems, they provide a wide vista for the 
examination of technical risk management in action. 

The F/A-18 Strike Fighter . Designed for Navy and 
Marine Corps use, has two engines, "redundant flight 
control computers and a backup mechanical flight control 
system" (55:1). Such redundancies, besides enhancing 
reliability, are also intended to enhance the survivability 
of the aircraft during a combat mission. The software 
development for the F/A-18 flight control system was behind 
schedule because Navy and contractor personnel 
underestimated the amount of work necessary. Also, 
requirement changes, driven by the need to correct a 
roll—rate problem, adversely affected the software effort. 
Further, "almost all of the flight control computer's 
memory space had been used while demands for additional 
space continued" (55:8). 

The use of composite materials on some F/A-18 
surfaces, instead of aluminum and titanium, resulted in 
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added technical risk. Specifically, poor predictions were 
made regarding the flexibility of composite structures 
since the aircraft industry was less familiar with 
composite structures than with aluminum and titanium (55:8- 
11). To enhance maintenance, ~ built-in test system was 
being developed for the F/A-18, however, as of 1980 a 
relatively high number of built-in test false alarms had 
been seen (55:22). 

The F/A-18 development and test effort was not without 
catastrophes because . . in September 1900 a development 
F/A-18 aircraft crashed in England because of a failure in 
the low pressure turbine [disk] of one of its F404 engines" 
(55:ii). Also, "on November 14, 1980, during an 
operational test and evaluation . . . (an F/A-18] aircraft 
entered into a spin while practicing combat maneuvers, and 
the pilot was unable to regain control" (55:ii). 

Chernobyl Nuclear Reactor Accident . In early 1968, 
the reactor core of a "graphitemoderated, water-cooled 
nuclear reactor" (63:2) exploded near Chernobyl in the 
Soviet Union killing 31 people and spreading deadly 
radiation. The Chernobyl reactor accident was attributed 
to "a combination of human error and poor reactor design" 
(63:8). To assess the risk that some United States 
reactors could fail the same way, some comparisons were 
made between some United States reactors and the Chernobyl 
reactor. In particular, a GAO study found that the 
Chernobyl reactor and the Fort St. Vrain (United States) 
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reactor both "a graphite core to control the rate of 
fission within the reactor" (63:2). However, the Chernobyl 
design allowed only seconds of response time to avoid the 
equivalent accident. This difference in allowable response 
time is due to the difference in the cooling system design 
in the two reactors (63:2,3,33; 83:39-42). The Fort St. 
Vrain reactor was specifically "designed so that the 
graphite in the reactor core absorbs most of the heat" 
(63:3). That heat is then transferred to helium gas that 
circulates through the graphite. However, in the Chernobyl 
reactor, the graphite was specifically designed only for 
fission moderation and not especially for heat absorption. 
Instead, the Chernobyl reactor uses cooling water to absorb 
the heat. Therefore, "the consequences of a loss-of- 
coolant accident would be more severe at a Chernobyl-type 
reactor than at Fort St. Vrain" (63:3). According to a 
Soviet report, the Chernobyl accident occurred while the 
reactor operators were conducting a test which required 
that a number of reactor safety systems be disabled 
(63:32). However, the Chernobyl reactor 

. . . v/as not designed to withstand loss of coolant 

without the use of the emergency cooling system. If 
the cooling is inadequate for a very short period 
(seconds) . . . the fuel would begin to melt. This 

would result in over-pressurization of the fuel tubes 
and explosions. (63:21) 

It is very clear that the hazardous events happend 
with such rapidity that past a certain point the Soviet 
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technicians could not respond quickly enough because "in 
less than 20 seconds, the reactor power level increased to 
100 times its designated power level" (63:32). In fact, 
"the power surge was so rapid that neither the control rods 
nor the coolant could be adjusted fast enough to stop the 
accident" (63:33). 

Airplane Development . On 17 December 1903 at about 
10:35 in the morning near K’tty Hawk, North Carolina, 
Orville Wright—with his brother Wilbur and five witnesses 
on the ground—successfully flew the first airplane. 

However Professor John D. Anderson, chairman of the 
Department of Aerospace Engineering at the University of 
Maryland, states that 

Contrary to some popular belief, the Wright Brothers 
did not invent the airplane; rather they represent the 
fruition of a century of prior aeronautical research 
and development. The time was ripe for the attainment 
of powered flight . . . The Wright Brothers' ingenuity 

dedication and persistence . . . [made them] first. 
(5:3) 

The efforts of the British inventor Sir George Cayley 
significantly advanced the theoretical understanding of 
aeronautics and was therefore instrumental in reducing some 
of the technological risk of attaining the 1903 flight with 
the Wright Brothers' aircraft. In 1804 Cayley built a 
revolving-arm mechanism which he used for experimenting 
with airfoils. Also, in 1804, Cayley built and flew a 
small (one-meter long) model glider. In 1849 Cayley built 
and flew a full scale glider that had three wings mounted 
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above each other. On a number of occasions, a small boy on 
the glider was lifted several meters into the air when the 
glider was traveling down a hill (5:8,9,10; 47:219; 

109:397). 

Other aeronautical experiments, such as the more than 
2,500 successful flights that Otto Lillienthal made in his 
hang gliders, clearly reduced the technical risk. On a 9 
August 1896 flight, Lillienthal crashed, broke his back, 
and died the next day. His last words were (translated from 
German): "Sacrifices must be made" (142:43). The Wright 

Brothers followed the Lillienthal hang glider flight 
experiments intently through written accounts and some of 
Lillienthal's reports (5:18). It was Lillienthal who first 
showed that curved wing surfaces were superior to flat wing 
surfaces for gliding (43:242). Wilbur Wright stated that 
"It was the death of Lillenthal that brought the subject 
(the problem of flight] to our attention" (io8:7). This 
spurred the Wright Brothers to write to the Smithsonian 
Institution in order to obtain books relating to flight. 
Orville Wright contended that he and his brother Wilbur had 
determined "that Lillenthal had been killed through his 
inability to balance his machine in the air" (103:7) . 
Further, the achievements of Samuel Langley and Octave 
Chanute also advanced the theory of flight (142:46). In 
fact, the sheer number as well as the content of the 
letters between the Wright Brothers and Octave Chanute 
indicates that Chanute was both a consultant mentor and 
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friend of the Wright Brothers up to his death in 1910 
(108:7-537; 109:677-993). 

Clearly, many technical risks associated with powered 
heavier-than-air flight were reduced in the century prior 
to the successful Wright Brothers flight. However, 
formidable technical risks remained for the Wright Brothers 
to solve. The Wright Brothers "took up gliding as a hobby 
while they were operating a bicycle repair shop in Dayton, 
Ohio" (46:146). A perusal of the content of the papers 
(e.g., letters and diaries of the Wright Brothers) reveals 
that from the beginning that they were meticulous in their 
study of the available literature relating to flight and in 
surveying the results of previous flight experiments 
(108:5,19,20-22). 

During 1900 and 1901 the Wright Brothers tested 
gliders in the steady winds of Kitthawk, North Carolina. 

At first these gliders were flown like box kites, later 
they were flown manned—usually pushed from a small hill. 
Wilbur Wright's notebooks are a testament to the methodical 
and meticulous nature of the Wright Brothers investigations 
into flight. His September to October 1900 notebook 
entries contains detailed observations that he made 
regarding the flight of birds. In particular, the flight 
characteristics of different types of birds were compared. 
Wilbur Wright made the simple observation that "no birds 
soar in a calm" (108:7) and indicated that he was paying 
particular attention to the equilibrium (i.e., balance) of 
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birds in flight (108:34-36). In a letter dated 23 
September 1900 to his father Bishop Milton Wright, Wilbur 
Wright fundamentally outlined the technical risk approach 
regarding the nearly finished first Wright glider: 

My idea is to experiment and practice with a view to 
solving the problem of equilibrium . . . once a 
machine is under proper control under all conditions, 
the motor problem [to obtain true flight] will be 
quickly solved. (108:26) 

That some letters indicated that Wilbur Wright gave 
priority to equilibrium in flight because once equilibrium 
was attained "a failure of motor will simply mean a slow 
descent and safe landing instead of a disastrous fall" 
(108:26). Further risk reduction was indicated by Wilbur's 
intention to conduct all experiments relatively close to 
the ground and to build his plane "to sustain five times my 
weight and am testing every piece" (108:26). Perhaps the 
most telling indication of the Wright Brothers' 
understanding of risk is the statement by Wilbur in that 23 
September 1900 letter: 

The man who wishes to keep at a problem long enough to 
really learn anything positively must not take 
dangerous risks. Carelessness and overconfidence are 
usually more dangerous than deliberately accepted 
risks. (108:26) 

Therefore, just in case of any potential crash landing 
Wilbur Wright concluded that landing on the sand near Kitty 
Hawk would lessen the risk of injury (108:26). Because the 
gliders initially exhibited little lift, the Wright 
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Brothers were convinced that the curvature of the wing 
surfaces was wrong. To address this technical problem they 
built a six-foot-long wind tunrel to study various curved 
wing surfaces in moving air. ^hey tested hundreds of 
glider wings in the wind tunnel and made "the first 
reliable tables of air pressures on wings" (46:146). This 
effort along with their quantitative records and analyses 
of their glides contributed immensely to their airplane 
design knowledge (108:119-170). In 1902, based on the 
result of their wind tunnel experiments, the Wright 
Brothers built a third glider with wings of 32 feet long 
and approximately 5 feet across. This glider was 
successful and allowed the Wright Brothers to make over a 
thousand manned glider flights—some for longer than one 
minute. At the time there were only crude propellers with 
efficiencies that were below 50 percent, and no developed 
theory for the design of an aerodynamic propeller. The 
Wright Brothers were "the first to recognize that the 
propeller is basically a rotating wing, made up of airfoil 
sections" (5:364). This understanding allowed them to 
design a propeller that was 70 percent efficient, and 
greatly contributed to the success of their first flight 
(5:3-23,363,364; 46:146-147). 

The development of an efficient propeller was not the 
only technological risk that the Wright Brothers had to 
overcome. Because no automobile engine existed that could 
meet their combined power and weight requirements as a 


71 




power plant for their aircraft, they found it necessary to 
develop their own using an automobile engine as a model to 
aid their design. The Wright Brothers' mechanic Charles 
Taylor was responsible for the detail design of the engine 
(114:56). In February 1903, during the first test of their 
engine, the aluminum crankcase cracked. Two months later, 
a replacement crank-case finished casting and in May 1903— 
roughly 7 months before their historic powered flight— 
their four cylinder (in-line) engine successfully passed 
its tests (5:367; 43:237; 114:55). 

Rocket Development . In July 1914, Goddard obtained 
"patents on rocket combustion chambers, nozzles, [and] 
propellant feed systems" (5*372). In 1915, Robert H. 
Goddard demonstrated the principles of rocket propulsion in 
a vacuum at Clark University in Massachusetts. In November 
of 1923, Goddard achieved successful operation of a rocket 
motor in a test stand using liquid oxygen and gasoline 
jointly as the rocket fuel. However, there was clearly 
technical risk since Goddard's early attempts to launch a 
liquid-fueled rocket were unsuccessful. Further, it was 
necessary for Goddard to develop specialized high pressure 
pumps to force the fuel into the rocket chamber. Finally, 
on 16 March 1926, Goddard successfully launched the world's 
first liquid-fueled rocket engine. This success in the 
field of rocketry was analogous to the Wright Brothers' 
first successful flight of the airplane at Kittyhawk, North 
Carolina in December 1903 (5:1,3,372; 42:4,16,17,21; 
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145:48). Robert H. Goddard's development of the rocket by 
transforming all the preceding theoretical rocket knowledge 
into actual practice, has been called "one of the most 
amazing lone-wolf development programs in the history of 
technology" (145:48). 

Almost continuously from 1930 to 1941, Goddard 
conducted development and flight tests at Rosswell, New 
Mexico with his wife and four assistants.. According to 
Goddard's wife, the "reliability of propulsion, stability 
in flight, and recovery, [of the rocket after launch], were 
the primary aims in the early tests, rather than the 
attainment of altitude" (145:46). However, to further 
reduce technical risks connected with rockets, particularly 
as it regards reliability, extensive mathematical 
reliability modeling had to be employed. 

The early development of mathc tical reliability 
models began during WWII in Germany, where a group led 
by Wern.ir von Braun was developinc; the VI missile. The 
first series of ten missiles was totally unreliable; 
they all blew up on the launching pads or fell into 
the English Channel. (89:1) 

This underscores the point that ir order to minimize 
technical risks of new technology development programs, 
special analytical or mathematical techniques may have to 
be refined or developed, and then applied (89:2). A brief 
chronology of the development of thr. rocket is provided in 
Appendix D. A detailed chronology cf Robert H. Goddard's 
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flight tests and results can be found in Von Braun and 
Ordwav's History of Rocketry and Space Travel (145:48-52). 

Atomic Bomb . The critical impact of experts in the 
reduction of technical risk is readily apparent in 
connection with the Manhattan Project—the United States' 
project that resulted in the world's first atomic bomb. In 
1939 Albert Einstein, at the urging of a number of 
scientists, sent a letter to President Franklin D. 

Roosevelt which expressed concern that Nazi Germany could 
be making significant strides towards the development of 
the first atomic weapon (41:397). President Roosevelt 
established the Office of Scientific Research and 
Development, and appointed the United States scientist 
Vannevar Bush as the director. In December 1941, the 
United States entered World War II. In May 1942, "The 
decision [was] made to proceed on all promising production 
methods" (45:325) for obtaining fissionable materials. 
Vannevar Bush decided to involve the Army in the 
construction of the plants that would be used to produce 
fissionable materials. Therefore in 1942, the Army Corps 
of Engineer opened a New York City office called the 
Manhattan Engineer District Office under General Leslie 
Groves. However, on 1 December 1942, when General Groves 
signe* the contract for the construction of a facility to 
produce plutonium, there were still many unknowns. Most 
notably, "the diffusion barrier (used to separate gases] 
had not yet been proven practical [and] plutonium 
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chemistry was almoat unknown" (45:325). Although a nuclear 
fission reaction had been achieved on 2 December 1942 in a 
nuclear reactor at the University of Chicago, formidable 
technical obstacles remained in the path of atom bomb 
development. One of the major technical challenges was to 
produce enough plutonium and to separate Uranium-235 (U- 
235) from Uranium-238 (U-238). A laboratory at Los Alamos 
(New Mexico), for exploring the methods of manufacturing 
the fissionable materials for an atomic bomb was placed 
under the direction of J. Robert Oppenheimer (46:517; 
47:831). 

Another major technical concerns regarded the approach 
that should be pursued in the atom bomb assembly. During 
the fall and summer of 1943, the "gun" approach was being 
pursued for the design of a nuclear weapon. Under the gun 
approach, "a mass of Uranium-235 or (Plutonium 239) would 
be fired into . . . another piece of Uranium-235" (45:325). 

When the two pieces joined they would become since 
"r.sutrons would be created at a faster rate than they can 
escape from the assembly" (45:324) resulting in an 
explosive release of energy (45:324-325). In April 1943, 
Seth Neddermeyer, a physicist under J. Robert Oppenheimer, 
"proposed to assemble a supercritical mass from many 
different directions" (45:325) instead of the two 
directions used in the gun approach. Neddermeyer's 
proposal—the implosion method—was supported by United 
States physicist Edward Teller and United States 
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mathematician John von Neumann. In July 1944 it was 
concluded that the implosion method was the appropriate 
one. However, there were significant technical hurdles to 
overcome in order to implement the implosion method 
(45:325). In particular, 

The big problem facing the physicists at Los Alamos 
[New Mexico] was how to produce an extremely fast 
reaction in a small amount of Uranium isotope, U-235, 
or of plutonium so that a great amount of energy would 
be explosively released. (81:180) 

John von Neumann, a theoretical mathematician who also 
had a crucial knowledge of hydrodynamics, was brought in as 
a con—to the group at Los Alamos late in 1943. Von 
Neumann's contributions ultimately resulted in the success 
of the sophisticated technique of attaining critical mass 
of the bomb's nuclear material by subjecting such material 
to a spherical shock wave. Further, von Neumann and James 
L. Tuck invented "an ingenious type of high explosive lens 
that could be used to make a spherical wave" (81:181). 
However, von Neumann's greatest contribution towards 
reducing the technical risk of atom bomb development was 
"his showing the theoretical people how to model their 
phenomena mathematically and then to solve the resulting 
equations numerically" (81:181) on the computer (42:44; 

81:178-181) . 

Indeed technical risks associated with developing the 
bomb were also significantly reduced by the emergence of 
the first electronic computer—the Electronic Numerical 
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Integrator and Computer (ENIAC). A number of crucial 
computations for atom bomb development were done with the 
ENIAC, and other computers, with the aid of von Neumann 
(81:117,150) . 

Another project in which von Neumann was involved 
resulted in the now famous Monte Carlo method. 
Fundamentally, the Monte Carlo method involves the modeling 
of a system as a repeated probabilistic experiment in order 
to evaluate the collective effect of probability 
distributions of the individi 1 components of the system. 
Von Neumann used the Monte Carlo Method for studying the 
diffusion and multiplication of neutrons. This reduction 
of the complex neutron interaction to a relatively simple 
Monte Carlo simulation gave the Los Alamos scientists a 
valuable tool for experimentation (78:652-653; 81:294-296). 

The fissionable materials were manufactured mainly at 
Oak Ridge, Tennessee and at Richland near Hanford, 
Washington, and then shipped to the Manhattan Project 
research laboratory at Los Alamos, New Mexico (46:517; 
47:831). A brief chronology of events regarding the 
development of the Atomic Bomb is presented in Appendix E. 

Laser Development . In .917 Albert Einstein recognized 
the principle of stimulated emission. The principle of 
microwave amplification by stimulated emission of radiation 
(MASER) was discovered by Charles H. Townes of Columbia 
University. In 1954 a working maser device was built by 
Townes, James P. Gordon and H. J. Zeiger, and subsequent 
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improved models were built by Bell Laboratories. Since the 
masers operated at microwave frequencies, Arthur L. 

Schawlow and Charles H. Townes proposed a technique to 
obtain a maser-like device which operated at optical 
frequencies. Assuming that the technical risk could be 
overcome, the device in question would in actuality be 
based on light amplification by stimulated emission of 
radiation (LASER). The first successful construction and 
operation of a laser was achieved in 1960 by T. H. Maiman 
of the Hughes Aircraft Company using a ruby crystal 
(44:686; 134:234; 135:227). 

Schawlow states that the building of an optical maser 
(i.e., a laser) "required preparation of an active medium 
that would actually display maser action in the optical 
region of the spectrum" (135:228). One of the major 
problems in using crystals or glasses as the lasing medium 
is that they "are usually formed at high temperatures and 
require considerable effort and expense" (101:255) to free 
them of optical imperfections. The ruby crystal in 
Maiman's laser had been machined into a rod whose ends were 
"polished optically flat and parallel and . . . partially 

silvered" (135:228). To obtain lasing, the ruby rod was 
then subjected to light from an electronic flash tube which 
could attain the necessary critical intensity (44:686). A 
brief chronology of laser development is presented in 
Appendix F. 
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Project Apollo . On 25 May 1961, President John F. 
Kennedy declared that the United States "should commit 


itself to achieving the goal, before this decade is out, of 
landing a man on the moon and returning him safely to 
earth" (116:736). A prerequisite for achieving the goal 
was the development of a booster—the Saturn V—capable of 
the required thrust (91:38). 

Saturn v Rocket . The three-stage Saturn V rocket 
was the product of highly experienced rocket experts who 
"had lost a number of rockets in earlier development 
programs and had blown up a few engines" (90:52). The 
rocket expert Werner von Braun had accumulated data from 
launching V-2 missiles in the New Mexico desert and then 
began building improved successors to the V-2. Other engine 
experts such as David E. Aldrich of Rocketkdyne, who "had 
helped to develop the engine for the X-2 research aircraft" 
(90:52), provided necessary expertise for the Saturn V 
development. According to Aldrich, since "the heart of (a 
jetj engine" (90 : 53) iv: the fuel injector, that is the 
first obstacle to successful engine development. Aldrich 
was the RocKetdyne program manager for the Saturn V F-l 
engine. An early technical problem with the F-l engine was 
that the fuel combustion "was so violent that it triggered 
shock waves producing more heat than the engine's cooling 
system could handle" (90:53). On 28 June 1962 one F-l 
engine was destroyed when to the "fire in the (combustion] 
chamber burned through the injector" (90:53). Within a 
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span of one and a half years, thirty different injector 
designs were tried without success. Finally, the test 
engineers decided to use explosive charges within the 
coirbustion chamber to trigger shock waves when the rocket 
was firing. It was thereby discovered that when baffles 
were used to isolate each combustion region, the shock 
waves in the chamber would die out. Other refinements such 
as enlarging the diameters of the holes from which the fuel 
and oxidizer were squirted, finally resulted in a solution 
(90:52,53). 

The size and weignt of the Saturn V was a major 
challenge since it stands "363 feet tall (and when) fully 
fueled it weighs nearly 6.4 million tons" (145:171). New 
building techniques had to be used for the Saturn V, and 
facilities capable of withstanding the weight of the 
tooling were necessary. One of the major trade-off 
decisions that had to be made for Project Apollo was thut 
of structure versus weight. The need to reduce the weight 
became so critical that the second stage of the Saturn V 
"emerged as the merest eggshell of a rocket" (90:99). This 
fact contributed to the rupture of two Saturn second stages 
during test (90:99; 145:170-173). 

The first launch of a Saturn V on 9 November 1966 was 
successful. Due to a decision by George E. Mueller, 
director of NASA's Office of Manned Space Flight, live 
upper stages for the Saturn V were used on these unmanned 
(i.e., test) launches. On the second launch in April 1968, 
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Two of the J-2 engines in the second stage shut down 
prematurely. The J-2 engine in the third stage failed 
to re-start on command, leaving the stage stranded in 
orbit. (90:60) 

These failures during the second Saturn V launch were 
of major concern because the next launch had been scheduled 
to be manned. The malfunction investigation of the second 
flight was confounded by the fact that the failed hardware 
was inaccessible—either in orbit or burned up upon re¬ 
entry. The telemetry (i.e., transmitted data) that had 
been obtained from the prior to the malfunctions was scanty 
since there were a relatively small number of Saturn V data 
channels. Further, these particular in flight failures had 
never been encountered during ground tests. The program 
manager Paul Castenholtz and Marshall McClure had been 
trying to determine why the failures occurred in space but 
not on the ground. By repeatedly studying movies of J-2 
engines firing on the test stands they began to realize 
that the ice that formed on the fuel and oxygen lines 
during ground tests was highly significant. The hypothesis 
was that the ice actually dampened some severe vibrations 
and thereby prevented the fuel and oxygen lines from 
rupturing. Using the available telemetry data Castenholtz 
localized the problem to a special part of the engine. 

Then, eight of the suspect fuel lines were operated in a 
special test chamber which duplicated the firing of a J-2 
engine in the vacuum of space. All eight fuel lines 
ruptured. Since the ice that formed on the fuel lines 
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during ground tests did not form in space, the vibrations 
were not dampened in space. This was the reason that a 
fuel line could break "after a few minutes of vibration in 
space" (90:60). The problem was solved by substituting a 
"rigid stainless steel pipe" (90:60) for the flexible fuel 
lines. "Thousands of man-hours" (145:171) were required to 
identify and correct the stated malfunction. The cause of 
both J—2 engine failures during the second Saturn V launch 
was specifically attributed to the "fatigue failure of a 
small liquid hydrogen line" (145:240). 

To reduce the enormous technical risks associated with 
landing a man on the moon and returning him safely to 
earth, the United States implemented a series of 
spaceflights designated as Project Apollo. One of the 
major technical decisions involved the actual flight path 
and moon landing technique that would be followed in order 
to land on the moon and return. It was known that the 
flight profile, in turn, would govern the "configuration of 
the entire vehicle" (11:33). There were strong arguments 
between "NASA and the academic science community" (11:33) 
regarding this issue of the flight path and the moon 
landing technique. In late 1962 a technique called the 
Lunar Orbital Rendezvous (LOR) method was generally 
accepted. Under the LOR method, the spacecraft would orbit 
the moon upon arrival. Then, one section would detach from 
the orbiting spacecraft and land on the moon's surface. 

For the return, the detached section would blast off from 
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the moon's surface and rendezvous (join) with the orbiting 
portion of the spacecraft (11:33,35). 

The Apollo spacecraft consisted essentially of three 
parts. The Command Module (CM) contained crew controls and 
living space. The Service Module (SM) contained the 
propulsion system, fuel and other. The Service Module was 
connected to the Command Module but the SM was jettisoned 
prior to re-entering the earth. Finally, the Lunar Module 
(LM) had an ascent and descent stage that was used to carry 
the astronauts between the CM, wh:ch remained in lunar 
orbit, and the lunar surface. All of this sat atop the 
Saturn V rocket during launch (91:40). 

Due to heavy schedule pressure, NASA management 
decided to do "all-up" flight testing. Instead of testing 
component by component, gradually building confidence in 
the system, the full apparatus would be tested in ready-to- 
fly configuration. This all up approach was considered to 
be unorthodox engineering (11:35). 

Gemini Space Vehicles . The Apollo program had 
been preceded by the considerable, but less ambitious. 
United States space endeavors such as Project Mercury which 
was the first United States manned space program, and 
Project Gemini (91:80-82,65-67). Project Gemini was the 
successor to Project Mercury. The Gemini series of space 
vehicles (12 in the series) provided the astronauts and 
ground crew valuable experience in activity (i.e., 
spacewalks), rendezvous and docking, long duration flights, 
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and guided re-entry. This experience was to prove 
invaluable (91:65). In fact, the first in-space emergency 
occurred during Gemini 8. On 16 March 1966 the Gemini 8 
spacecraft with Armstrong and Scott achieved the first in¬ 
space docking with an Agena target rocket placed in orbit 
101 minutes earlier" (91:66). After linking, the two 
vehicles started "tumbling and spinning out of control" 
(91:66) due to a jammed Gemini thruster. Armstrong and 
Scott "escaped only by firing their retro-rockets" (91:65) 
and returned to earth two days earlier than planned. Also, 
tragically, t.he original crew of the Gemini 9, See and 
Basset, died while trying to land "a jet fighter in bad 
weather at McDonnell works" in St. Louis (91:67). 

The Apollo program started off with tragedy 

. . . when on 27 Jan 1967 a short circuit in the 

electrical systems set fire to the all-oxygen 
atmosphere while the crew, Virgil Grissom, Edward 
White and Roger Chaffee, were carrying out a full 
launch rehearsal on Pad 34. White, reaching back over 
his head was unable to open the hatch in the few 
seconds before the crew was overwhelmed. (92:53) 

In the wake of the tragedy, major re-design of the 
space-craft cabin and marked improvements to the crew 
operations were undertaken to greatly minimize crew safety 
risks (77:71,271; 92:53). 

Site Selection . To reduce the enormous technical and 
operational risks involved in achieving a roundtrip manned 
moon landing, each flight in the Apollo series tackled 
increasingly difficult technical and operational 
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milestones. Some highlights of selected Apollo flights are 
presented below. Technical teams, especially geologists, 
devoted 5 years looking for a suitable lunar landing site. 
Several of the flights in the Apollo series, particularly 
those which orbited the moon, had an objective of observing 
possible lunar landing sites. Such detailed surveillance 
and selection of potential lunar landing sites was 
necessary to curtail the risk that the moon lander would 
slide off a cliff, or land on uneven ground and overturn. 

It was further necessary to verify that the surface was 
firm enough so that the lander would not sink and be 
swallowed up upon landing. Some of the stated technical 
risks connected with the lunar surface were addressed by 
the series of Ranger and Surveyor spacecrafts. Rangers 1 
and 2 were orbited around the Earth "to check out 
instrumentation" (145:191). Rangers 3 through 6, intended 
to explore the moon, failed, but in 1964 and 1965, Rangers 
7, 8 and 9 relayed some 17,000 images" (150:44) of the 
moon's surface before crash landing onto the moon. The 
pictures, with resolution of 1 foot, showed rocks on the 
moon's surface and indicated that the surface could support 
some heavy objects (145:191; 150:43,44). 

However, site selection, had to take into account 
other risk factors such as "communication, tracking, fuel 
capacity, rocket performance" (150:44). When the Soviet 
Union's Luna 9 and the United States' Surveyor I probes 
both landed on the moon in 1966, the moon's surface was 
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"barely dented" (150:45). Follow-on Surveyor probes 
established the properties and chemical composition of the 
lunar surface. Once it was determined in what areas to 
concentrate the search for landing sites, a series oi five 
Boeing-built Lunar Orbiters were sent to photograph the 
lunar surface so that by February 1968, five potential 
lunar landing sites were selected by relying on the 
expertise of geologists. The top two candidate sites were 
then observed by astronauts in Apollo 8 and 10 during lunar 
orbits as low as 10 miles above the lunar surface. The 
information obtained from the Surveyor and Orbiter probes 
was crucial for the planning of the Apollo manned mission 
(145:193; 150:46,47). 

Simulator Training . While the geologists were 
searching for a safe and suitable landing site, the 
astronauts were training for the actual moon landing 
operations. In order that the astronauts could simulate 
landing on the moon, a Lunar Landing Training Vehicle 
(LLTV) was built. The LLTV was a truss assembly (no wings) 
built around a vertically placed jet engine. The thrust 
allowed vertical takeoff and landing. When the LLTV thrust 
was adjusted to support five-sixth of the vehicle weight, 
the LLTV could be used to effectively simulate the effect 
of landing in lunar gravity from about 500 feet above the 
moon's surface. "To prepare the astronauts for 
emergencies, the LLTV was occasionally programmed to fail" 
(150:44). There were a number of occasions when an LLTV 
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pilot had to eject from a malfunctioning LLTV. When the 
attitude control rockets—which kept the flight stable— 
shut down putting the LLTV into a spin, Neil Armstrong 
ejected at 200 feet. Another NASA pilot had to eject 
during another attitude control failure, while yet another 
pilot ejected from an LLTV that was blown out of control by 
strong winds (150:44). 

Apollo 7 . The October 1968 Apollo 7 flight, 
although confined to Earth orbit, was the first manned 
flight of the Command and Service vehicles. During Apollo 
7, the main propulsion engines of the Service module were 
fired automatically as well as manually, and the engine 
performance was evaluated. Further, the heat shield of the 
vehicle was evaluated during re-entry (92:53). 

Apollo 8 . While the Apollo 7 operations were 
confined to Earth orbit, the December 1968 Apollo 8 flight 
was man's first flight around the moon. In fact, it was 
the first time that the 3000 ton Saturn 5 rocket had been 
used to launch men into space. During the 147-hcur Apollo 
8 mission, the crew achieved the critical lunar orbit 
insertion. They spent 20 hours circling the moon, and 
photographed potential landing areas (92:53; 149:1), 

Apollo 9 . Apollo 9 continued the trend of 
incrementally reducing the technical risks associated with 
landing a man on the mocn. The March 1969 Apollo 9 flight 
was confined to Earth orbit, however, it was the first 
manned flight with the Lu<»ar Module. During the ten day 
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flight, separation, rendezvous and docking was practiced 
between the Lunar Module (LM) and the Command Module (CM). 
Although the flight was confined to earth orbit, some 
operations were nevertheless performed to simulate a moon 
lift-off (92:53). Apollo 9 the May 1969 Apollo 10 flight 
was "a successful dress rehearsal for the Moon landing 
[that occurred] two months later" (92:53). During the 
Apollo 10 flight, the entire Apollo spacecraft orbited the 
moon for the first time. To simulate a moon landing, the 
Command Module maintained a 111 kilometer orbit around the 
moon, while the Lunar Module, with Stafford and Cernan on 
board, separated and descended twice to nearly 14.5 
kilometers from the surface of the moon. The Lunar Module 
then re-docked with the Command Module after a separation 
of 8 hours (92:53). 

Apollo li . Finally, the July 1969 Apollo 11 
flight, with the crew of Neil Armstrong, Edwin Aldrin and 
Mike Collins, resulted in the first men m the moon 
(148:1). However, despite the previous Apollo flights, 
there remained significant sources of risk that were 
evident during the Apollo 11 flight. For instance, on the 
way down to the moon in the Lunar Module "Armstrong decided 
to take over manual control because the spacecraft was 
approaching an area in the Sea of Tranquility strewn with 
boulders" (92:54). In fact, "had Armstrong not taken 
control (the craft] might have overturned or smashed on 
alighting" (116:765). Other technical risk was evident 





since it was confirmed that one of the on-board computers 
was over-loading. That computer "was a vital part of the 
radar system which calculated the LM's [Lunar Module's] 
attitude and rate of descent, and on which the life of the 
crew depended" (92:54). 

Although the Apollo 11 flight put the first men on the 
moon on 21 July 1969, there were nevertheless near 
disasters on some of the later moon landing flights. This 
clearly illustrated that the sources of technical risk were 
not eliminated. For instance, the November 1969 Apollo 12 
mission 


. . . started sensationally, for as the launch was 

being made through a rain squall, the Saturn 5 rocket 
was struck by lightning. The spacecraft's electrical 
system was briefly put out of action and for the first 
time during a manned launch they were very close tc an 
abort. (92:55) 


But, the flight was not aborted and the crew members 
of Apollo 12 went on to complete a mission that resulted in 
31.5 hours of walking on the moon (92:55). Appendix G 
provides a brief chronology of the achievement of a man on 
the moon. 

Apollo 13 Accident . The April 1970 Apollo 13 mission 
had been planned to be the third moon landing mission. 
However, 


. . . an explosion on board when the spacecraft was 

329,915 km from the earth all but cost the lives of 
the crew and turned the mission into a 3.5 day rescue 
drama ... as tens of thousands of technicians worked 
to bring the crippled spacecraft safely home. (92:56) 
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Initially, it was determined that due to the explosion 
only half the power, water, and oxygen that was needed to 
get the crew home, would be available. However, based on 
simulations that were done by technicians on the ground, 
some ways were found to conserve energy by powering down 
systems. This unfortunately meant that the cabin became 
increasingly cold and disaster threatened again . . when 

the astronauts, tired and chilled, made mistakes" (92:56). 
For the journey home, it was necessary for the astronauts 
to move into the Lunar Module and use it like their 
"lifeboat (a backup craft for their safe return) . . . 

using a jury-rigged arrangement ... to purge carbon 
dioxide from their atmosphere" (92:56). It was later 
determined that an oxygen tank had overheated and exploded 
since some "heater switches had welded shut when subjected 
to excessive pre-launch electric currents" (92:57). The 
exploding tank took another oxygen tank out of commission 
as well. Astronaut Lovell—one of the Apollo 13 crew 
members—concluded that 

. . . warning signs during testing went unheeded, and 

the tank, damaged from 8 hours of overheating, was a 
potential bomb the next time it was filled with 
oxygen. That bomb exploded on 13 April 1970—200,000 
miles from Earth. (92:57) 

This illustrates that testing—intended to decrease 
technical risk—if done incorrectly, can actually increase 
technical risk by causing damage (92:56,57; 147:1). 
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Space Shuttle Challenger Disaster . Inadequately 
managed technical risk has resulted in disasters and 
national defense setbacks. The in-flight explosion of the 
Space Shuttle Challenger (Flight 51-L) on 28 January 1986, 
resulted in the death of seven astronauts on board the 
shuttle, and the concomitant loss of part of the military 
space operations capability. Investigations revealed that 
the hot exhaust gases in a solid rocket motor of the space 
shuttle eroded, and then bypassed two "O-ring" seals. 

These hot gases eventually ignited the hydrogen in the 
nearby space shuttle External Tank, thereby causing an 
explosion. However, the potential for just such a 
disastrous O-ring sealing failure on the solid rocket 
motors had been identified as early as 17 December 1982 and 
the failure potential of the seals was under study at the 
time of the disaster. Indeed, Morton Thiokol, Inc., 
producer of the solid rocket motors, had recognized the 
need for improvement of the field joints as well as the 
motor case/nozzle joint prior to the accident (68:3; 
102:38). In the wake of the Challenger disaster, nearly 
220, 155, 31 and 8 modifications were made, respectively, 
to the Orbiter, solid rocket motors, main engines and 
external tank. In particular, a third "0-rir.g" and a metal 
seal feature was added to the solid rocket motor field 
joint (12:1; 67:10,11; 93:24; 102:38,39; 103:24-26). 

Some of the major changes that were made to the Space 
Shuttle solid rocket motors to reduce technical risk were 
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actually quite mundane, yet very important. These included 
the following: 

Heaters to maintain seal temperature at 75 degrees 
Fahrenheit or above, a weather seal to prevent water 
entry into the joint and possible freeze-up, longer 
pins [used to hold the rocket segments together at 
each joint] and new retention bands, an alternative 
insulation (J-seal) to eliminate the putty previously 
used. (67:11) 

In addition to the hardware changes that were made, a 
number of procedural changes were recommended by the 
Presidential commission that investigated the accident. 

The commission recommended that detailed testing of the 
solid rocket motor joint be conducted in the "exact flight 
configuration in the vertical position" (67:10), and that 
the National Research Council oversee the NASA re-design of 
the solid rocket booster or joints (67:10-13). 

Other commission recommendations included an increased 
emphasis on rotating some former astronauts into management 
positions. Further, the commission recommended that a 
major effort be undertaken to provide a crew escape 
capability and an improved in-flight launch abort 
capability for the space shuttle. Most of the recommended 
changes were implemented (67:32). Thirty-two months after 
the Challenger disaster, the shuttle named Discovery, with 
five astronauts aboard, was flown to a successful mission 
(93:20). 

After the Challenger disaster, NASA intensified their 
three-step risk analysis effort. The NASA three-step risk 
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analysis effort consists of a Failure Modes and Effects 
Analysis (FMEA), a Critical Items List and a Hazard 
Analysis. The FMEA is conducted to identify hardware that 
are "critical to performance and safety of the vehicle and 
mission" (67:8). As the second step, NASA makes a Critical 
Items List (CIL) which identifies those components whose 
failure alone (i.e., a single failure) would cause either 
loss of life, vehicle or mission. The CIL contains the 
rationale for those specific components in tha design that 
do not have redundancy. Such rationale should provide 
"information regarding the (1) design, (2) tests 
accomplished to assure integrity of the hardware, (3) 
specific inspection points, and (4) operational means to 
mitigate the failure" (67:8). The CIL is based on the 
results of the FMEA. As the third step, NASA conducts a 
Hazard Analysis (HA). The purpose of the HA is to identify 
and recommend necessary corrective actions for those 
potential sources of risk that arise during the operation 
and maintenance of the system hardware and software. The 
HA also examines sources of risk arising from man/machine 
interfaces (e.g., the crew), the environment, and 
mission—related activities (67:7-9). 

Inertial Upper Stage Anomaly . Three years before the 
Challenger disaster, on 5 April 1983, hot exhaust gases 
were also cited for eroding and thereby bypassing seals on 
the Solid Rocket Motor #2 (SRM-2) of the Inertial Upper 
Stage (XUS) booster. This ultimately resulted in a mission 


93 





anomaly whereby a NASA satellite—the Tracking Data Relay 
Satellite (TDRS)—was placed in an incorrect orbit. The 
combined TDRS satellite and the IUS booster had been 
released from the payload bay of the Space Shuttle in low 
Earth orbit (less than 200 miles above Earth). During this 
sixth space shuttle mission the IUS was supposed to boost 
the satellite to the much higher geosynchronous orbit. 
However, the combined satellite and IUS booster began to 
tumble in space about 83 seconds after the start of the 
SRM-2 burn. Specifically the Inertial Upper Stage Anomaly 
Board investigation concluded that the failure resulted in 
an IUS "nozzle gimble mechanical failure" (4:4) that 
resulted in the SRM-2 nozzle being jammed in an off-center 
position. Due to the mechanical failure the nozzle 
positioning actuators were unable to correctly reposition 
the nozzle until the SRM-2 motor burned out (i.e., finished 
thrusting). The author of this thesis was in the Inertial 
Upper Stage Systems Program Office at the time (4:4). 
However, the engineers were later able to successfully 
execute a plan to command the satellite to fire its small 
rocket thrusters and finally achieve the correct orbit 
(4:4; 120:110). 

In fact, the technical risk from "flame erosion" 
(66:39) caused by the leakage of hot gases has even been 
manifest after post-firing inspections of the Phoenix 
missile in April, 1986 (66:39). 
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AFOS Software Development . A 1982 GAO report cited 
that the National Weather Service's automated data 
processing and telecommunications system was undergoing 
some major technical, scheduling and managerial problems 
relating primarily to software development. The system is 
called the Automation of Field Operations and Services 
(AFOS). One of the problems cited was that the "AFOS 
hardware lacks sufficient core memory to accommodate 
current software or applications initially planned for 
AFOS; the Weather Service cannot tell how much more memory 
is neede" (65: ii)- In addition, the operating system 
could not "meet concurrent processing requirements 
originally specified" (65:ii). 

The GAO cited that one of the major problems was that 
the software was "unnecessarily complex" (65:31). For 
example, each of the software modules in the AFOS were made 
so interdependent that even minor changes to one module 
caused major technical repercussions to the other modules. 
This module interdependence also complicated the 
troubleshooting of programs. In addition, the AFOS system 
would experience "deadlock when the computer (attempted] to 
process two or more tasks that need the same resources" 
(65:31). Also, information that was being input would be 
lost if the system malfunctioned during the inputting 
(65:31,32). One of the primary causes of problems with the 
National Weather Service (NWS) software was NWS 
inexperience with software projects of that magnitude. Due 
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to inexperience, the NSW had erroneously assigned a higher 
priority to the data processing hardware than to the 
software development procedures (65:31). In fact, the GAO 
found that "development priority was not clearly 
established and top priorities [were] ignored in favor of 
low priority work" (65:21). 

Changing user requirements meant that completed 
software work had to be redone. A large amount of 
technical risk also stemmed from inadequate design 
specifications. With only ambiguous specifications to 
guide them, each programmmer tended to act independently 
rather than as part of a coordinated team. An outgrowth of 
this was that the software documentation so "critical to 
effective software development and operation" (65:35) was 
clearly deficient (65:31-35). 

SEa ce , Defense Initiative (SDI). 

The Space Defense Initiative (SDI) is a "program to 
determine the feasibility of developing and deploying a 
defense against nuclear ballistic missiles" (72:1). To 
reduce the inherent technical risks, the SDI organization 

has used ... 3 parallel contractor efforts to 
analyze technology, propose preliminary system 
concepts and associated mission performance and 
interface requirements, define data needs for sensor 
design, and develop preliminary demonstration plans. 
(72:4) 

The technical risk reduction strategy include;, the 
conduct of some "technology validation experiments" (71:1) 
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by the Army's Strategic Defense Command. The objective is 
to demonstrate that issues regarding critical technologies 
are resolved so that the risks in any ensuing full scale 
development phase are also reduced. For instance, to 
ensure that effective sensors for a strategic defense 
system can in fact be developed, the Army had initially 
pursued two competing sensor designs. In fact, it has been 
determined that three types of sensors—corresponding to 
the three trajectory phases of launch, midcourse and 
terminal—would be necessary. Technical risk has also been 
identified in connection with the design of a sophisticated 
data processor and with the air turbulence expected during 
airborne tests of sensors (6:94? 71:12). Key technologies 
have been identified for SDI such as 

1. component technologies for long-wave infrared 
sensors including optics, detectors, signal 
processors, and cryogenic coolers, technologies 
for laser 

2. technologies for laser radars and space-based 
radars. (72:3) 

However, it is clear that a host of experimental or new 
technologies may eventually be necessary to achieve an 
actual strategic defense. As a result, the Livermore 
National Laboratory is carrying out X-ray laser research in 
support of SDI (69:1,2). It is possible that space nuclear 
rocket technology will become very important for SDI 
applications (64:8,11). The SDI program is also highly 
reliant on analytical capability consisting of 
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supercomputers, software and analysts, in order to test and 
evaluate a number of potential strategic concepts and 
systems for battle management and command and control. As 
of early 1988, there was difficulty in achieving a timely, 
secure (i.e., cryptographic) supercomputer link between 
three key analytical sites (70:1-5,9). 

National Aero-Space Plane 

The National Aero-Space Plane (NASP) program was 
established in December 1985 as a joint Department of 
Defense and NASA program. The program involves such 
Department of Defense agencies as the Air Force, Navy, 

DARPA, and SDIO, (60:1-10) and was established as a 
"development and demonstration program" (60:22). The NASP 
has been tentatively designated as the X-30 since it is "an 
experimental vehicle and not a prototype or operational 
vehicle" (60:27). 

To reduce the technical risks in building the 
prospective X-30, the technical community has identified 
"critical or enabling technologies" (60:10) whose 
availability and maturity should be monitored. Such 
technologies include advanced strength, refractory (i.e., 
high melting point) materials. To gauge the technical risk 
to the NASP, engineers have compared the anticipated f’ight 
trajectory of the NASP to that of the Space Shuttle. bile 
the Space Shuttle "reaches orbit very quickly in an almost 
vertical flight trajectory, the X-30 would achieve speeds 
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of Mach 25 in the upper atmosphere before [achieving] 
orbit" (60:13). Further, the X-30 would take off 
horizc tally and be essentially air breathing while the 
Space Shuttle does neither. However, the re-entry 
trajectory for the X-30 is expected to be the same as the 
Space Shuttle (19:67-68? 60:12-15). 

The anticipated X-30 trajectory has also been compared 
to that of existing aircraft. Thus it is expected that the 
"X—30 must be designed to fly ten times faster and higher 
than existing air-breathing aircraft" (60:12). The X-30 is 
therefore expected to spur greater emphasis on hydrogen 
fuel technology (19:71? 60:12,15,40). 

According to NASP and NASA scientists, the greatest 
technical risk connected with the X-30 is "the development 
of an air-breathing propulsion system" (60:25). However, 
the development of suitable materials and the integrating 
of the X-30 1 s major subsystems such as the thermal control, 
structures, propulsion and avionics, also provide 
formidable technical risks. Indications are that the X-30 
design and development will require significant 
computational resources. In particular, there is a known 
requirement for "computational fluid dynamics to predict 
the aerodynamic, thermal and propulsion characteristics at 
. . . (Mach 8 to 25)" (19:71? 60:25; 122:14). 

The NASP program office has formulated a strategy for 
reducing the technical risks. One of their groundrules is 
that existing national facilities (e.g., wind tunnels and 




laboratories) will be used where possible to preclude 
delays due to construction of new facilities (60:26). 
Indeed, it has been contemplated that a suitably 
instrumented space shuttle could be used to aid the X-30 
design. Another groundrule is that more than one technical 
approach will be pursued for high risk components or 
systems to increase the chances of finding appropriate 
solutions in a timely manner. Consequently, the NASP 
program intention is to compete a number of contractors 
with the expectation that each contractor will pursue 
different—perhaps even highly different—conceptual 
approaches. Simultaneously, the NASP program intends to 
promote efforts to advance those technologies that are 
critical to the X-30. Additionally, a series of 
milestones, with corresponding design reviews and key 
decision points, have been established in order to 
facilitate the reduction of technical risks (60:26,35-40). 

The plan is to minimize safety risks connected with 
the X-30 by incorporating the appropriate safety features 
into the design and operation of the vehicle. For example, 
the X-30's propulsion engine will have at least two 
engines, and the flight control system is planned to have 
four backups. Some features of Ine vehicle's operation 
would also contribute to safety. The flight trajectory is 
such that the vehicle would largely fly above adverse 
weather (60:45-47). In the event of an aborted mission, 
the vehicle will be able to maneuver and make a powered 
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Figure 6. Technological Trend of Increasing Aircraft Skin Temperature (140:68) 
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Figure 7. Technological Trend of Increasing Engine Temperature (140:68) 











landing. The use of the intended hydrogen fuel has been 
determined to pose less danger than conventional fuels 
"since its ignition temperature is 1,065 degrees Fahrenheit 
or twice that of aviation grade kerosene" (60:46). 

However, the technical strategy has taken into account 
other sources of technical risk. There is a risk of 
foreign object damage to the X—30 since "small rocks on the 
runway, hail, ice, rain, or even space debris" (60:46) 
could pose considerable danger primarily to the X-30's 
engines and skin. Some peripheral technical concerns have 
also been considered such as the need for new fuel 
processing and handling procedures (60:46-48,77,78). 

Summary of Case Studies 

The case studies reveal that the sources of technical 
risk vary and may be unanticipated. For the F/A-18 
aircraft, the technical risk was due to new materials 
(composites) that were used, the underestimated need for 
computer memory capacity, and certain in-flight failures 
that led to crashes. For the Chernobyl Reactor, human 
error and poor design introduced technical risk. For 
airplane development, the Wright Brothers addressed 
technical risk by carefully determining the cause of 
Lil1 ienthal's fatal crash and then identifying and 
concentrating on solving what they determined to be crucial 
for achieving successful flight. Specifically, the Wright 
Brothers gave the solving of the flight equilibrium problem 
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priority over the development of a suitable engine or 
propeller. To develop the rocket, Robert Goddard worked to 
first understand such critical rocket components as rocket 
nozzles, combustion chamber and propellant feed system and 
then conducted numerous captive and flight tests of 
rockets. The Atomic Bomb development required not only the 
achievement of unprecedented production (i.e., separation) 
techniques for plutonium and uranium isotopes, but also new 
analytical techniques (e.g., Monte Carlo method) and 
computational tools (i.e., the electronic digital 
computer). Laser development required an extrapolation 
from maser principles and the overcoming of difficulties in 
the preparation of an appropriate optical material. 

The Apollo 11 landing of a man on the moon required 
that certain formidable prerequisites be satisfied such as 
the development of a high thrust vehicle (the Saturn V), 
selection of the flight path and moon landing sites, and 
astronaut training. The technical risks were fundamentally 
overcome in stages via successive space flights leading up 
to the Apollo 11 moon landing. The Apollo 13 Accident, the 
Space Shuttle Challenger Disaster, the Inertial Upper Stage 
Anomaly and the AFOS Software problems clearly illustrate 
how inadequately managed technical risks can result in 
significant program problems, setbacks or disasters. 

The case studies indicate some important potential 
"unknown unknowns"—factors which constitute unanticipated 
sources of technical risk. One potential unknown unknown 
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is the need for one or more secondary development efforts. 
For example, since no adequate engine or propellor was 
available to power an airplane the Wright Brothers had to 
develop both. Similarly, after initial launch failures, 
Goddard had to develop specialized high pressure pumps to 
force the fuel into the rocket chamber. The Atomic Bomb 
case study shows, one unknown unknown is the need for 
relatively unprecedented analytical techniques and tools. 
Figure 8 outlines various types of potential "Unknown 
Unknowns." 
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"UNKNOWN UNKNOWNS" 
REVEALED BY THE SELECTED 
CASE STUDIES 





















IV. 


Analysis Considerations 

Decision making must often be made in the presence of 
risk. To reduce technical risk, correct and adequate risk 
assessments must be made by technical experts. Then 
follow-on risk analyses allow effective decisions to be 
made by identifying those alternatives that have the 
highest probability of success. According to Klopp, a 
successful decision (i.e., risk analysis), requires: 

1. A problem to be solved 

2. Viable and achievable alternative courses of 
action 

3. Information 

4. A decision maker 

5.. A decision strategy. (95:107) 

Timson has presented the general model for system 
development decision making shown in Figure 9. Timson's 
model illustrates the potentially cyclical nature of system 
evaluation and revision. Similarly, General Thurman 
emphasized that "... the planning for uncertainty and 
risk turns out to be a dynamic and iterative process" 
(143:15). Further, the technical and performance risk is 
heightened because . . complex performance requirements 

[of multiple subsystems or components] . . . must interface 

perfectly. . . (143:12). 

Prevention . Where possible, technical risk should be 
prevented. The way to achieve this is to select proven 
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rigure 9. Model of System Development Decision 
Making (144:32) 









techniques, materials, and designs which involve very 
little uncertainty. However, a Rand report has stated that 
Air Force missions often require increasingly higher 
technology since "the newer systems must perform more 
functions . . . with greater precision . . . [with] more 
integration among functions" (129:23). This implies that 
technical risk is often unavoidable. 

The prevention of technical risk is especially crucial 
where safety is concerned. Accordingly, MIL-STD-882B 
System Safety Program Requirements states that the first 
safety precedence is to eliminate hazards by an appropriate 
design. Thus the first design priority is, if at all 
possible, to design a system such that technical risk is 
eliminated . However, the total elimination of all 
technical risk in a particular system is rarely attainable. 
Therefore the second design priority is to minimize 
technical risk by appropriate design practices (3:113; 

39:6). 

Uncertainty as the source Risk . Technical risk in 
complex weapon programs stems from uncertainties connected 
with "... inadequate knowledge of the basic technology or 
its specific implementation" (143:12). There is the 
initial pitfall that the actual technical problem being 
evaluated will be formulated incorrectly and will thereby 
result in the right answer to the wrong question (126:15). 
For a risk reduction plan to be effective, it must lead to 
successively more information which ultimately identifies 
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and reduces both the anticipated and unanticipated 
technical unknowns connected with a project. Klopp states 
that there is a "continuum of knowledge" (80:148) ranging 
between having no information and total information on the 
internal and external factors that govern the technical 
success of a particular system. Clearly, few good 
decisions will be made when there is no information. 
Instead, actions are often undertaken to obtain sufficient 
information prior to a decision. Therefore, the most 
important task of any decision maker is to identify where 
there is uncertainty due to incomplete information and 
attempt to resolve that discrepancy (80:148). Kraemer has 
proposed the risk assessment model of Figure 10 which 
results in a quantified risk assessment (i.e., 
probabilities). In particular, Kraemer has divided 
problems into "normal risk" and "higher risk" in his model. 
Henley and Kumamoto proposed the program risk assessment 
framework of Figure 11 which begins with a qualitative 
assessment of risk, that is then quantified and ultimately 
results in a risk management strategy. 

Pitfalls . Although the general source of technical 
risk is uncertainty, more specific factors that contribute 
to technical risk are 

1. Underestimation of the degree of the technological 
breakthrough required in a state-of-art of product 
development, while under a fixed and tight time 
and budget constraint 

2. Pushing technology too fast 

3. Lack of prototype development 
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Figure 11. Henley and Kumamoto's Risk Assessment Framework 

(89:14) 
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4. Performance requirements beyond state-of-the-art 

5. Inadequate, test program 

6. Major design or scope changes. (132:35) 

These stated factors can become pitfalls if they ate 
overlooked. The case studies of the IUS and the AFOS 
illustrate that technical over-optimism leads to an 
underestimation of the technical risks involved. To ensure 
that the most realistic assessment of the technical risks 
are obtained as soon as possible, the system should be 
prototyped as soon as practical. Early prototyping of the 
system is especially important if the in-house experience 
level of the technical managers and designers is more 
theoretical than "hands-on". Early prototyping has been 
cited. 

state-of-the-Art . Because technological risk is 
determined, in large part by the state-of-the-art, Rowe and 
Somers have proposed the factors cited in Figure 12 for 
evaluating the technological state-of-the-art (2'’•2). In 
general, the higher the state-of-the-art (i.e., the newer 
the technology), the greater is the technological risk 
involved. Accordingly, greater emphasis must be placed on 
early accurate risk assessment. 

The technical success of certain development efforts 
often hinges upon advancemei cs in a few critical areas. 

For instance, advancement in ". . . propulsion has led the 

way for all major advancements in flight velocities" 
(5:327). Therefore, accurate and timely identification 
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1. Size—number of interrelated components, physical volume 

2. Complexity—difficulty in meeting performance requirement 

3. Newness of technology—experimental state of technology 
i 4. Percent proven technology—degree of newness 

5. Experience in the field-work on similar programs 

6. Percent new components—test and evaluation requirement- 

7. Interdependency of subsystems—types of linkages 

8. Degree of precision—quality or cleanliness requirements 

9. Special resources—testing or tooling requirements 

10. Definitive specifications—clarity in meeting requirements 

11. Design flexibility—tolerance level, substitutes available 

12. Required theoretical analysis—need to support proposed design 

13. Degree difference from existing technology—life cycle of 
technology 

14. Available knowledge in the field—amount of experimentation 
required 

15. Infra-structure support required—degree of dependency on 
vendors 


Figure 12. Factors Determining the 
State-of-the-Art (132:40) 
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of those high risk factors and components which are 
critical to correct system performance is mandatory for 
reducing technical risk. Once identified, these critical 
technical risk areas must be prioritized by experts and 
emphasized accordingly. 

However, the technical success of one system may also 
be dependent upon the technical success of one or more 
separate, but allied, system or systems. For example, the 
GAO had determined that the "Anti-Submarine Warfare 
Standoff Weapon is subject to increased risk due to 
problems in a related program" (58:8). Technical risk also 
arises due to excessive interdependency between subsystems. 
The AFOS case study illustrated that if software modules 
are made highly interdependent, then a modification to one 
module will have repercussions to other software module and 
the system will be immensely complex to modify, 
troubleshoot or update. Therefore the interdependency of 
subsystems should be minimized unless absolutely necessary. 

Expert Resources . To avoid or reduce technical risk, 
it is critical "that the proper specialists are asked the 
right questions at the right time and that their advice is 
heeded when given" (137:83). Technical development case 
histories illustrate, and Shannon stresses, that the 
violation of this fundamental concept in part or in total, 
is responsible for most technical failures. 

Timing . Based on lessons that were learned in DOD 
acquisition it is well established that 
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. . . the concept phase is the critical phase for risk 
and uncertainty analyses; and that an innovative 
tailored approach is required for each program . . . 
the analysis at concept phase establishes the basis 
for the rest of the program by developing system 
specifications and technology base. (143:21) 

Risk assessment and analysis is needed most during the 
concept definition phase—the earliest phase—of a 
development program, because it is here that technical 
uncertainty is greatest. In the concept definition phase, 
there are usually a number of general technical approaches 
that appear viable. The correct choice from among these 
approaches, along with successively correct choices within 
the selected approaches (e.g., design, materials and 
electronic components) is significantly aided by performing 
a risk assessment (24:V-11? 143:21). Further, it is 
critical that the risk assessment be available for 
decisions during the concept studies since "by the end of 
concept studies, 70 percent of the key decisions on the 
weapon system have been made" (128:138) according to Lt Gen 
Reynolds, formerly Vice Commander of Air Force Logistics 
Command. 

Technical experts are required as early as possible to 
identify and select those design alternatives which will 
preclude as much technical risk as possible. Their 
expertise then becomes critical for minimizing risk in the 
design. However, because a "risk assessment is the first 
of the steps in a risk analysis" (10:12), the risk 
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assessment must be accurate or all the ensuing effort will 
be invalid to some degree. 

Planning for Technical Contingencie s. The 
technological uncertainty inherent in a new technology 
development program means that the "real world occurrence 
of technological breakthroughs, and catastrophes" (144:93), 
particularly the latter, must be realistically considered. 
Specifically, the responsible agency must ensure that ", . 

. historical safety data, including lessons-learned from 
other systems are considered and used" (39:4) to improve 
the contemplated or existing design of the system. Such 
effective design techniques such as simplicity, modularity, 
redundancy of subsystems, safety and design margins, etc., 
should be used to minimize the technical risk by design. 

Among other things, the case studies illustrate that 
especially where new technology is involved, established 
analytical methods, analytical methods, test techniques, 
and facilities may be wholly or partially in adequate or 
invalid. Thus the success of the Atomic Bomb development 
was to a large extent dependent on Dr. John von Neumann's 
modeling of critical mass phenomena on one of the first 
electronic computers. It is probable that without either 
the contributions of Dr. von Neumann or the opportune 
emergence of the electronic computer, the successful 
development of the atomic bomb would have been impossible, 
or severely delayed. Similarly, it has been determined 
that the design and development of the National Aerospac • 
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Plane (X-30) would require the use of computational fluid 
dynamics because "wind tunnels are of no use in modeling 
speeds above Mach 8" (122:14). In general, the case 
studies show that there are some identifiable categories of 
technical risk that are usually unanticipated. Some of 
these unanticipated categories of technical risks—unknown 
unknowns—were outlined above in Figure 8. 

Analysis of Interviews . The interviews were conducted 
in accordance with the methodology of Chapter II. 

Appendices K through P contains transcripts of most 
interviews. The WRDC interviewees were virtually unanimous 
in stating that technical risk reduction is not the first 
priority of the laboratories. Rather the primary focus in 
the laboratories is in exploring the technical 
possibilities. For example, Dr. Olsen, Chief Scientist of 
the WRDC Flight Dynamics Laboratory, stated that "Risk is 
something that I don't think that we (the labs] manage 
much. We generally work on trying to prove that something 
is feasible and can be done. . . ." (119). However, the 
scientists stated that the laboratories are implicitly 
involved in technical risk reduction every day since the 
labs must ultimately transition sufficiently mature (i.e., 
low risk) technologies to the System Program Offices. 
According to Dr, Keith Richey, the Technical Director of 
WRDC, the labs . . make a conscious decision as to when 
to turn it [the technology] loose" (130) since they must 
transition the technologies relatively fast to the Air 
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Force at sufficiently low technical risk. According to Dr. 
Richey, there will rarely be zero risk in a technology that 
is transitioned to the SPOs since the labs cannot examine 
all possible applications of a particular technology (130). 

Dr. Richey stated that acquiring ". . .as much 
knowledge as you can have on the physical and non-physical 
phenomena that you are dealing with" (130) is the most 
important technique for managing technical risk. Similarly 
Dr. Karris Burte, Chief Scientist of the WRDC Materials 
Laboratory, emphasized that "it is important to identify 
what you know and what you don't know and that what is what 
is not known is probably the most important source of 
technical risk" (14). According to Dr. Arnold Mayer, Chief 
Assistant for Research in the Vehicle Subsystems Division, 
analysis, mathematical and physical models, tests, and 
demonstrations, are used in the iterative process of 
investigating feasibility and that this equates to 
investigating technical risk (106). Mr. Jack Cannon, ASD's 
Technical Director for Development Planning, stated that 
"if you had to select one, then certainly prototype testing 
is the way you reduce risk" (17). Similarly, Dr. u; 
Brown, Chief of WRDC's Technology Assessment Division, 
stated that a suitable demonstration (e.g., flight 
demonstration) is essential (13). It became very clear 
from the interviews that in Dr. Olsen's words: "... risk 
in the laboratories is not the same as risk in a SPO" 

( 119 )• 


119 





Model Review Comments 


The preliminary model was revised to incorporate 
comments of the panel of experts. Some specific written 
comments from their reviews may be found in Appendix J. 
However, many of the comments that were provided were 
written on the actual copy of the preliminary model. In 
addition, information obtained during the interviewing of 
the experts was also incorporated into the model. 

The model had initially indicated that the System 
Requirements Specification is written just prior to the 
concept exploration. However, Dr. Squire Brown's comment 
was that it would most likely be a "needs" statement at 
that stage. He also indicated in his review that, in his 
words, "there may be an intermediate stage before 
prototype—a technical demonstrator like the X-29, followed 
by the Advanced Tactical prototype." The model was revised 
to incorporate these comments (under Stage 3. and Stage 7 of 
the model, respectively). Because Dr. Brown also expressed 
that some transitions in the model were unclear, he was 
provided a later subsequent version of the model for an 
additional review. 

Dr. Jess Riles 1 first model review comment (Appendix 
I) was the question "Where does cost enter into the model?" 
Although the scope of this research (as stated under Scope, 
Limitations, and Assumptions in Chapter I) does not 
expressly address costs. Dr. Riles' rationale that 
"Technical feasibility may be demonstrated but may not be 
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affordable," is compelling. Therefore, a (prerequisite) 
block was added under Stage 1 of the model that requires 
that one "Identify and Confirm Funding Constraints." His 
other two comments were in accord with the research 
groundrule that some technical feasibility of the 
development program in question has been demonstrated. 

After Dr. Squire Brown's second review of the evolving 
model, he had some additional comments. Specifically, he 
recommended that the design priority sequence be corrected 
to show that the first design priority in Stage 8 is to 
achieve the primary performance objective of the system and 
only then is the rest of the design priority sequence 
valid. These comments were discussed by phone with Dr. 
Brown and then incorporated into the model. 

Model Presentation 

The synthesized risk management model in divided into 
tan stages and can be found later in this chapter beginning 
on page 131. Citations are provided to document the 
specific source whereby each respective model component was 
derived. Many of the model components have multiple 
citations where necessary to fully document the model 
synthesis. Some blocks in the model indicate pertinent 
technical risk management examples. These specific 
examples are in lower case letters and enclosed within 
bracket; all other citations pertaining to the block are 
placed outside the bracket. For example, on the first page 
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of the model, the Atomic Bomb is listed in lower case as an 
example in the block entitled "Identification of a Real or 
Potential Threat." The citation pertaining to the Atom 
Bomb example is placed within the brackets. 

Logic Symbols . The model uses or and and logic 
symbols or "gates" as shown in Figure 13. For the or gate, 
a correct output occurs at D only if input A or input B or 
input C is present, or if all three inputs are present 
simultaneously as inputs to the or gate. 

For the and gate in Figure 13, a correct output is 
obtained at D only if all inputs to the and gate, inputs A 
and B and C, are present. If any of the indicated inputs 
to a particular AND gate are missing then no (valid) output 
will be obtained at the output of that particular and gate. 
Thus, the and gate is literally being used as a "gate" in 
the model that ensures that the correct inputs are obtained 
before any output is passed on as an input to the next 
succeeding task. 

Main Line . A key feature of the model is the Main 
Line. This is the thick, black line on the left hand side 
of every page of the model. The risk management tasks flow 
along the Main Line subject to the fulfillment of 
prerequisites. These prerequisites are indicated by blocks 
that are off of the Main Line but which are generally 
inputs to AND gates along the Main Line. For ease of 
reference, all blocks along the Main Line, except those in 
Stage 8 of the model, are numbered sequentially. The Main 


122 








Line blocks in Stage 8 are distinct due to their "priority" 
assignment. 

Design Diamonds . The model contains four diamond¬ 
shaped blocks which indicate that specific "Yes/No" 
question is asked and the corresponding path, analogous to 
a decision, is then followed in the model. 

Requirement Block . The model actually starts with the 
Requirement Block. The OR gate that precedes the 
Requirement Block has four inputs: (the requirement may be 
mandated, (2) resulting from a deficiency, (3) stem from a 
threat, or (4) be due to the decision to pursue a 
particularly mission-enhancing capability. After the 
Requirement Block, the model is divided into Stage 1 
through Stage 10. 

Stage 1 . After the requirement has been specified 
and authorized, we proceed to block 1 where we "Identify 
Appropriate Technical Disciplines and and Experts ..." 
that are pertinent to the specific requirement. However, 
this identification of experts can be correct only if the 
appropriate representatives from the using organization, 
and laboratory experts, and individuals with previous 
experience, that are pertinent to the requirement, have 
been identified. 

Stage 2 . Similarly, we should proceed to Stage 2, 
the Initial Study by Experts, only if the three identified 
prerequisites for the first step in Stage 2 are satisfied. 
During this stage the requirement is confirmed and 
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communicated to the experts) along with any salient 
priorities within the requirement that the using 
organization has identified. This way when the cadre of 
(national) experts—of the correct diversity, depth, and 
breadth of expertise—have hopefully studied the 
requirement somewhat prior to their assembly. It is 
therefore possible that the experts may arrive at the first 
formal assembly having already identified some of the "Key 
Variables Governing Successful Achievement of the 
Requirements" or with an understanding of various technical 
tradeoffs that may eventually have to be investigated prior 
to the system design (Blocks 4 and 5). For example, for a 
rocket booster as the Project Apollo case study made clear, 
there is a trade-off between the weight (e.g., structural 
strength) and the payload carrying capability. 

At Block 6, the broad technical objectives of the 
system are initially specified. These objectives should be 
consistent with the operational requirements from the using 
organization. At Block 7, the system's technical 
objectives are initially prioritized although this may have 
to be revised based on results of forthcoming analyses or 
prototyping. For an effective prioritization, one 
prerequisite is specific previous experience, by the 
individuals doing the prioritization, is mandatory. The 
other immediate prerequisite shown for Block 7 is a network 
of the technical tasks (e.g., a PERT network) would be 
essential for showing the interrelationship among the tasks 
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and for ultimately determining the potential impact of late 
completion of technical tasks on the entire system 
development schedule. At Block 8, a systems requirements 
statement—much more general than a specification—is 
written. 

Stage 3 . During Stage 3, the concept exploration 
and technical risk assessments are conducted as Blocks 9 
and 10 show. The pitfall identified in conjunction with 
Block 9 is that technical overoptimism must be guarded 
against to ensure that no potential weaknesses or problems 
with a contemplated technical concept are overlooked. 

Eight prerequisite tasks are shown before the Block 10 
Technical risk Assessment task can be considered completed. 
For instance, the "level of the technical expertise that is 
needed vs available" must be evaluated. In general, the 
technical risk assessment will have to be done for each 
concept that is examined. In Block 11, the technical risks 
corresponding to each concept are prioritized (i.e., rank 
ordered). 

Stage 4 . In Stage 4, specifically Block 12, a 
concerted effort is made to identify and plan to resolve 
the important unknowns that have a bearing on the outcome 
of the system development effort. In order to achieve the 
Block 12 task, one prerequisite is for resolving 
information uncertainty is to "Identify Relevant National 
Superlabs, Expertise, . . . ." and existing simulations or 

software that can help to resolve any information 
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uncertainty relevant to the technical effort. This stage 
is also important because it may prevent unnecessary 
duplication of tasks that have already been done elsewhere. 

Stage 5. In Stage 5, Block 13, there is a 
concentrated effort to acknowledge a plan for "what-ifs"— 
to consider courses of action prior to the actual 
occurrence of technical problems. In Block 14 through 16, 
criteria for evaluating candidate approaches are 
identified. The approaches (Alternatives) are then 
analyzed against the criteria and those approaches that 
meet or exceed the criteria are selected as candidate 
technical approaches. Finally, in Block 17, the 
alternatives which survive the screening are tentatively 
rank ordered for the most rigorous study in Stage 6. 

Stage 6 . In Stage 6 there is an intensive effort to 
ensure that a rapid leaning curve is achieved with respect 
to the surviving technical approaches and the potential 
technical risks. Per Block 18, the candidate concepts are 
investigated in detail with respect to the prerequisites 
identified: "Analytical and Simulation Tools, Math 

modeling, etc." In Block 19, in-depth simulation or 
physical models are used. For adequate risk reduction, one 
cannot proceed past Block 19 unless the applicable "worst 
case" parameter values as well as "most likely" parameter 
values have been identified for the design under study. 
Further, it must be ascertained that the simulation is 
representative of "real world" conditions in order for the 
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simulation output to be of value. As part of the learning 
curve process, the organization's technical knowledge must 
be continually enriched (Block 20). This enrichment can 
take the form of recruiting "fresh" technical personnel 
from time-to-time, training, and attendance at appropriate 
technical seminars. 

Stage 7 . In this stage the basic system design 
corresponding to each candidate approach is outlined (Block 
22) and key technical parameters for measuring technical 
progress of the prospective system are tentatively 
identified (Block 23). Some experimental prototyping of 
some components or subsystems that are anticipated to be 
high risk (Block 23) is pursued in this stage. Assuming 
that the indicated prerequisites are met, one or more 
Preliminary Design Reviews (Block 24) are held to select 
the primary technical approach for the system and/or for 
the system's key subsystems (Block 25). Block 26 is a 
decision diamond regarding whether or not the selected 
primary approach should be pursued solely. If the 
technical approach selected is not relatively old then we 
come off the "No" side of the diamond and pursue two or 
more approaches in parallel. If the technical approach is 
old than we come off the "Yes" side of the diamond. In 
either case we go through the OR gate to a decision diamond 
regarding the experience of the personnel (Block 28) in the 
selected approach or approaches. If the personnel are not 
experienced, then we rush to Stage 8, Design of Prototype." 
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block 29 likewise requires a decision on the technical and 
operational familiarity with the environments within which 
the system will operate. This results in selection of the 
Block 3OA or Block 30B tasks to determine how the system 
will function in the use environment. Block 31 involves 
the formation of specific strategy to minimize the risks 
involved with eventually achieving a successfully 
functioning prototype. Such a strategy can then be applied 
to actual prototype design (under Stage 8). At Block 32, 
experimental prototypes can be used in early field tests in 
order to obtain an early evaluation of suitability to the 
operational environment. 

Stage 8 . This stage involves the actual design 
and building of the prototype system. Stage 8 is divided 
into nine (9) design priorities ranging from designing to 
meet or exceed the primary performance objectives to 
designing for easy modification at subsystems. Thus, the 
highest design priority is Design Priority 1 and the lowest 
design priority is Design Priority 9. Collectively, these 
design priorities ensure that the higher priority design 
task is achieved before a lesser task is given emphasis. 

In practice these design priorities may be pursued somewhat 
simultaneously if the indicated prerequisites of each have 
been met. 

Stage 9 . Once the prototype has been designed and 
built , a technical risk evaluation of the actual system is 
made. The potential hazards that may arise from the system 
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(e.g., explosion, fire) are identified (Block 33). The 
Boeing checklist in Figure 4 can be used for this. The 
prototype must then be evaluated for possible failure modes 
and the consequences of those failures (Blocks 34 through 
38). The failure mode analyses along with information 
derived from testing, will eventually result in the 
implementation of design improvements. Then more Critical 
Design Reviews can be held to evaluate the technical 
progress and to decide on necessary design changes for 
subsequent (e.g., more mature versions) prototypes. 

Because the Critical Design Review occurs after a prototype 
has been designed and built instead of before , more 
technical information upon which to base a decision 
regarding necessary design modifications. 

Stage 10 . Detailed testing of the prototype (or a 
later version) is conducted in Stage 10 of the risk 
management model. "Specific Test Objectives" and 
"pass/fail criteria," along with the other items shown, are 
prerequisites for effective testing (block 41). At the 
decision diamond (Block 42) technical experts must be 
consulted (Block 43A) if there is a test failure and design 
changes must be implemented (Block 44). After the design 
changes there will be a retest. If the system passes, no 
further design changes are implemented (Block 43) and the 
readiness for production of the system is evaluated in 
accordance with Air Force System Command Regulation 84-2 on 
Production Readiness Reviews. 
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MODEL FOR TECHNICAL RISK MANAGEMENT 
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2. CO NFIRM'CCM MUNI CATE USER 
REQUIREMENTS AND PRIORITIES AMONG 
REQUIREMENTS 



4. IOENTIFY KEY VARIABLES GOVERNING SUCCESSFUL ACHIEVEMENT OF THE 
REQUIREMENTS (5:327; 81:180; 11:33) 


S. IDENTIFY INTERDEPENDENCIES AND TRADEOFFS AMONG VARIABLES |e.g. weight vs 
structural strength, flight profile vt vehicle configuration, design vs producibiity] 
(28:24.26. 85.217; 132:42) 
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8 WRITE SYSTEM REQUIREMENTS STATEMENT 
(28:24) 



STAGE 3; CONCEPT EXPLORATION / TECHNICAL RISK 


EXPERTS WITH CLEAR UNDERSTANDING OF 
LIMITATIONS OF EXISTING SYSTEM 
(107:59; 135 227) 


EXAMINE POSSIBILITY OF MODIFYING OFF-THE-SHELF 
HARDWARE AND TECHNOLOGY (104 27. 143:17) 


OBTAIN RELEVANT PREVIOUS RESEARCH 
(5:3,8-10,18 , 108:5,19,20-22) 
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EVALUATE FOR AVAILABILITY OF 
PREREQUISITE MATERIALS (19 52; 68; 
1 33:13. 1 35:227; 140:67 


REQUIREMENTS FOR LEVEL OF PROCESS 
QUALITY OR QUALITY OF WORKMANSHIP 
[e g , parts-per-billion purity required for 
scmi-conductors (18 9)] (66:4.28-30) 


PREREQUISITE TECHNOLOGIES [e g . 
Significant computer advances came wnn 
advent of the transistor vs vacuum lubes 
(18:6)| (97:6) 



10 CONDUCT TECHNICAL RISK ASSESSMENT OURING CONCEPT 
STUDY PHASE 
(85:217; 143:17.21) 




12 SPECIFY PLAN TO OBTAIN ADDITIONAL INFORMATION 
CORRESPONDING TO EACH ITEM OF TECHNICAL UNCERTAINTY 
(30:15; 150:43-47) 





















STAGE 5: INITIAL TECHNICAL CONTINGENCY 
PLANNING 



* IDENTIFY NATIONAL SUPER-EXPERTS AND 

FACILITIES WITH EXPERIENCE CORRESPONDING 
TO EACH POTENTIAL "UNKNOWN UNKNOWN" 
IDENTIFIED IN FIGURE 8 
(132:40. 137:83) 


’ IDENTIFY ANO PRIORITI2E POTENTIAL HAZARDS 
(89 21) 



J 1 3 PLAN FOR TECHNICAL CONTINGENCIES 
{3:51; 98 280. 143 17) 



14. ESTABLISH CRITERIA FOR SCREENING 
CANOIOATE ALTERNATIVE TECHNICAL 
APPROACHES 

(94.138-141; 125:3. 1.43:1 7) 


IS. ANALYZE ALTERNATIVES 
(29 6S. 143:21)_ 


16. IDENTIFY POTENTIALLY VIABLE 
TECHNICAL ALTERNATIVES / CONCEPTS 
BASED ON SCREENING CRITERIA 
(45:324.325) 















17. RANK ORDER ALTERNATIVES WHICH 
PASS THE SCREENING CRITERIA 
(94:142) 



STAGE 6: ACHIEVE FAST LEARNING CURVE 



MATH MODELING OF SYSTEM OF SYSTEM 
(81:181; 112:7; 143:18) 


PROBLEMS. LIMITATIONS OF PREDECESSOR OH 
ANALOGOUS SYSTEMS 
(39 4; 99:226; 108:7) 


IOENTIFY LIMITATIONS OF AVAILABLE 
COMPUTATIONAL TOOLS AND techniques 
(81:181.89:1) 
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29 . 



1 


30A. RECRUIT INDIVIDUALS WITH 
OETAILED KNOWLEDGE OF USE 
ENVIRONMENT, (e g , consult *ith 
astronauts regarding space) 
(82:263) 


30B. USE ANALYTICAL AND COMPUTER 
TECHNIQUES TO PREDICT THE EVENTUAL 
SYSTEM PERFORMANCE 
(82:262) 


MINIMIZE SUBSYSTEM OR MODULE 

INTERDEPENDENCY 

(65:31; 132:90) 


SIMPLICITY (MINIMIZE COMPLEXITY IN 
THE DESIGN) 

(29.39; 65:31) 



INCREMENTAL DEVELOPMENT AND 
DEMONSTRATION (e g . Each flight m the 
Project Apollo senes overcame technical 
obstacles to enable the Apollo 11 rr.an an 
the moon 
(91:53)1 (132:44) 
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STAGE 8: DESIGN OF PROTOTYPE 



KNOWN WORST CASE 
ENVIRONMENT 
(51:5-8, 73:13} 


















Pn_Of ij^_ 



3rd Design Priority 


DESIGN TO MINIMIZE RISK OF 
MALFUNCTION IN SYSTEM (39:6) 


r 


KNOWLEDGE OF POTENTIAL HUMAN- 
INDUCED MALFUNCTIONS 
(54:30. 62:8) 


IDENTIFY POTENTIAL HAZARDS AND 
HAZARD SEVERITY (39 6) 


REDUNDANT SUBSYSTEMS 
(SSI) 


RELIABILITY ANALYSES (Refer to MIL- 
STD-78SB(38) 


DESIGN/SAFETY MARGINS (e g., derating 
of electronic parts (28:77-79; 125 3) 


FAILURE INDICATION le g . During 
Apollo 11 moon landing, failure codes 
indicated that the computer was 
overloading (91:54)] 


PERSONAL PROTECTIVE DEVICES (39 G) 


REDUNDANT SUBSYSTEMS (ACTIVE OK 
PASSIVE REDUNDANCY) 

(55:1) 
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MODULARITY 
(29 43-45) 





STANDARD PARTS AND MATERIALS 
(NON-EXOTIC) (28:24) 



LOW INTERDEPENDENCY OF MODULES 

SUBSYSTEMS 

(65:31) 


AVOID PITFALL OF SOFTWARE OR 
SYSTEM DOCUMENTATION WHICH OOES 
NOT REFLECT CHANCES IN A TIMELY 
MANNER (65:35) 















STAGE 9: POST-DESIGN RISK ANALYSES 



33 IDENTIFY THE POTENTIAL HAZARDS 
(e g., Ar> explosion, fire, harmful release 
(89 19)1 













35 SPECIFY THE BOUNDARIES-LIMITATIONS OF THE 
HAZARD STUOY 

[e g Will effects Of war, 'abotaqe. acts of nature, 
etc oe considered?! 

(39 20) 



37 CONSEQUENCE ANALYSIS 
(8929) 



38 EVALUATE OESlGN FOR FAILURE MODES AND 
IDENTIFY CONSEQUENCES OF FAILURE 
i«J9 31) 


39 ASSIGN CRITICALITY OF FAILURE MODES 
(67 3. BO:31) 


40 CRITICAL DESIGN REVIEW PER APPENDIX E OF 
MIL-STD-15213 (28 32) 
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SOFTWARE AND OPERATIONAL TESTING 
BY INDEPENDENT AGENCY 
(56:38,39; 11213.15,16) 


SOFTWARE TEST TOOLS 
<56 10,12) 


AVOID PITFALL OF INTRODUCING DAMAGE 
OR RISK DUE TO INCORRECT,EXCESSIVE 
TESTING ((e g.. Incorrect pre-launch 
testing caused damage */hich resulted in 
explosion during Apollo 13 flight 
(91:57)) 


PRECLUDE POSSIBILITY OF OBTAINING 
SCANTY OR INVALID (TEST) DATA (e g 
Cause of failure of second Saturn V 
rocket test flight difficult to determine 
due to scanty telemetry (90: 60)] 

(141: 93) 























•n conouct 
TESTING (32:2) 
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V. Conclusions and Recommendations 

Overview 

Inadequately managed technical risks have resulted in 
setbacks, failures and operational disasters in Department 
of Defense programs. Therefore, the purpose of this 
research was to synthesize a model that epitomizes a 
strategy for the management of technical risk. The model 
was synthesized using the three-pronged effort of: (1) a 
literature search and review to determine what previous 
work was done in risk assessment and risk management, (2) 
case studies of historical, contemporary and prospective 
new technology development programs, and (3) interviews 
predominately with Chief Scientists at the Wright Research 
and Development Center (WRDC) at Wright-Patterson Air Force 
Base. The model validation was via reviews of the model 
that were made by the stated interviewees. If the model is 
used as a technical management guide and decision aid by 
individual program or project managers at all levels, 
collective marked improvement in the technical risk 
management throughout the Department of Defense may be 
achieved. 

Conclusions 

The research reveals that the management of technical 
risk is essentially a sequential, cumulative, repercussive 
and iterative activity. Technical risk management is 
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basically sequential because the start of one particular 
task often requires the successful completion of one or 
more preceding tasks; one task often requires inputs from 
one or more previous tasks. Technical risk is cumulative 
because failure to adequately fulfill the prerequisite 
tasks contributes to the technical risk for all succeeding 
tasks. For example, the act of selecting an inferior or 
uncertain design approach may thereafter incur significant, 
risk to successful system development regardless of whether 
or not all succeeding tasks are performed in exemplary 
fashion. Technical risk management is repercussive because 
changes to a particular component in a system may cause 
undesirable, or even unanticipated, technical ripples 
throughout the entire system. Therefore, to minimize 
system complexity and risk, it is important to understand, 
and if possible, limit the interdependence of modules and 
subsystems within a system. 

Finally, technical risk management is iterative 
because as the technical effort continues, knowledge gained 
about system deficiencies or potential for improvement 
results in changes to the system. Since changes occur 
during the development of virtually all systems, the 
technical approach must be flexible enough to accommodate 
system•revisions indicated by analysis, test, production, 
and operation. 

According to Mr. C. J. Cosenza, Chief of the Advanced 
Development Division of WRDC’s Technology Exploitation 
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Directorate, technical risk is usually managed in stages so 
that the risk is systematically less in successive stages 
(22). However, he emphasized that where time is the major 
factor, such as when a system is urgently needed to counter 
a known threat, then multiple tasks must proceed in 
parallel since technical risk considerations and cost 
become secondary to the schedule in that case (22). 

Model Value/.Validation 

The technical risk model has been reviewed by 
cognizant technical experts of WRDC. Appendix J contains 
written comments that were provided, on the preliminary 
model. Mr. Jack Cannon, Technical Director of the 
Development Planning in the ASD Office of Plans, said that 
the risk management model is valuable to the program 
manager who skips some of the intermediate steps to show 
him the "management risk he is taking ... by treating as 
unneces; ary certain steps" (Appendix J). That is, the 
model prevents the program manager from unknowingly 
skipping steps or being unmindful of the potential 
technical future risks he may be incurring to the 
development of a system by doing so. Dr. Jim Olsen, Chief 
Scientist of the WRDC Flight Dynamics Laboratory wrote that 
"the logic (in the model] is impeccable." He stated that 
the important technical "question/answers are not 
frequently addressed, or are addressed in an uncoordinated 
way" during systems development. Therefore, the technical 
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risk management odel may ensure that the crucial questions 
and answers have been systematically asked and obtained, 
respectively, prior to each important system decision to 
achieve technical risk reduction. The model was developed 
based upon inputs revised from an exclusive literature 
review, case analyses, and interviews with experts. 
Information was then synthesized into a realistic and 
workable out put the mode. 

After his second review of the evolving model. Dr. 
Brown remarked that the model is "an interesting piece of 
work" and that it is in accord with his experience in the 
area of technical risk assessment/management. Mr. Cannon's 
comment that "unknown unknowns" (i.e., unanticipated events 
that cause technical risk) cannot actually be evaluated. 
Therefore, the prerequisite regarding "unknown unknowns" 
(near Block 13) was clarified to mean the "potential 
unknown unknowns" that were derived from the case 
histories. 

Recommendations for Further Research 

Although the technical risk management model has been 
validated by a group of technical experts, it now requires 
formal testing in System Program Offices. 

DSMC Evaluation . The model should be provided to the 
Defense System Management College (DSMC) for their 
evaluation and possible incorporation into acquisition risk 
management strategy. Specifically, the model may be very 
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beneficial when used in conjunction with KIL-STD-1521B— 
essentially an effective technical checklist. Recently, 
the Willoughby templates (36) have been drafted as part of 
a strategy to attempt to decrease the Department of Defense 
risks in transitioning programs from development to 
production. The model herein encompasses technical risk in 
general and references the Willoughby templates via 
reference (36). The model may therefore provide a strong 
foundation for a general comprehensive technical risk 
management strategy. 

Subjective Evaluation . One method of doing this would 
be to provide the model as a technical risk management 
strategy and decision aid to ten (10) Department of Defense 
program managers (at the Captain to Lieutenant Colonel 
levels and civilian equivalents). This group will be 
designated as the "model group." After a definite time 
interval (e.g., 18 months) the group with the technical 
risk model should be asked to provide specific comments as 
to the utility benefit or shortcomings of the model. 

Testing . For actual testing, a control group of 
program managers, that will not receive the model, will be 
identified. The objective of the trial would be to 
determine whether the programs being managed by individuals 
in the model group achieve a superior technical status—an 
indication of better technical risk management—than the 
control group. Realistically, since the required period 
over which to evaluate technical progress could be long, 
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the comparison can be made by evaluating whether the model 
the control group are practicing a better technical risk 
assessment and management based on predetermined criteria. 

Specifically, at the end of the trial, participants 
would complete a survey questionnaire (Likert scale) which 
would attempt to evaluate to what extent they are following 
effective technical risk management techniques. For 
instance, they would be asked to respond on a Likert scale 
to what extent they have ensured or are personally aware 
that the following have been (or are being done) in their 
program: 

1. Specific laboratory experts that could help their 
program in a consulting capacity in the event of technical 
problems, have been identified. 

2 . Technical program risks have been identified. 

3. Technical program risks have been prioritized. 

4. Interface requirements have been (are being) 
specifically defined. 

5. Technical personnel connected with the program 
attend professional seminars and obtain suitable training. 

6. There is a capability to simulate or reproduce 
system malfunctions for study. 

7. Personnel from the using organization who can 
provide more information or details on the use environment 
and the specific employment of the system have been 
specifically identified. 
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8. Exotic materials, parts and processes of the 
system have been specifically identified. 

9. Backup options for exotic materials, parts and 
processes have been identified. 

10. The specific worst case locales where the system 
man be used (e.g., sandy, dusty, cold, hot, at altitude or 
a combination) have been specifically identified. 

The ten criteria just stated are a partial list of the 
criteria that could be used as indicators of how well the 
technical risks of a program are being managed. Although 
the responses to the questionnaire would not guarantee that 
a particular program manager is in fact managing technical 
risk as indicated by his/her responses, the responses 
should generally be a good measure of risk management. If 
the model is of high utility then theoretically the model 
group should score higher. 

Given the variation in program type, maturity, and 
external influences (e.g., budget, schedule changes) in the 
Department of Defense it may be necessary to select 
programs which are at or near the same Department of 
Defense acquisition milestone, or to ensure that the model 
and control groups have the same mix of programs at the 
various milestones. 

Management Implications to POD 

To successfully achieve the objectives of a particular 
Department of Defense weapon development program, the 
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program manager must effectively manage the program's risk. 
Since technical risk often has a direct bearing on the risk 
of failing to achieve both a target cost and schedule, it 
is essential that an effective strategy be formulated and 
implemented for managing risk. The importance of strategy 
has been previously discussed in Chapter I (Justification). 
However, a particularly noteworthy example of the potential 
advances that can be achieved by implementing an effective 
strategy is the emergence of Japan as a dominant of Japan's 
defeat in World War II. The Japanese have "attributed much 
of their success to the teachings of such Americans as [Dr. 
W. Edwards] Deming" (136:541) in the areas of quality and 
industrial management. The nation that can more ably 
manage technical risk is likely to have decided defense and 
industrial advantages over nations with lesser technical 
risk management skills. With a superior technical 
management strategy, a greater percentage of defense 
programs may be done correctly "the first time," and within 
budget and schedule. Further, a period of diminishing 
Department of Defense resources and a public that is 
increasingly intolerant of even minor defense acquisition 
fiascos, may require that an improved technical management 
strategy be implemented. 

It is likely that no general officer would propose 
that a war be fought without a potentially effective combat, 
strategy which pervades even the lowest echelons. By 
analogy, defense weapons development and acquisition could 
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benefit from a pervasive technical risk management strategy 
that is available even to lower level program and project 
managers. Such a deliberate strategy could supplant 
marginally effective—and perhaps impromptu—technical risk 
management approaches that may be in use by some Department 
of Defense program managers in the day-to-day management at 
the lower and middle level (officer) echelons. A codified 
technical risk management model is likely to enhance 
communication and synchronize the efforts of managers and 
engineers at all levels since that model would provide a 
common reference point. 

Ideally then, the model that resulted from this 
research could be provided as a reference to every program 
manager at. almost all levels. The technical risk 
management improvements which the model may facilitate for 
each individual program manager could result, collectively, 
in a quantum improvement of technical risk management in 
weapons acquisition throughout the Department of Defense. 
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Appendix B: Interview Questions 


1. What are your technical qualifications: Specifically 

what technical degrees do you hold? 

2. What is your technical experience? [What technical jobs 
have you held?] 

3. What is the mission of the organization of which you are 
a part? 

4. What is your present job? 

5. In your opinion, what are the most important techniques 
for managing technical risk? 

6. Based on your experience, what is the most important 
technique for managing technical risk? 

7. Does your organization have a specific written strategy 
or policy for assessing and managing technical risk 
(e.g., document, statement of policy, operating 
instruction)? 

8. Can I have a copy? 


After each interview had a copy of the preliminary model 
for at least six working days the following questions were 
asked: 


1. lave you reviewed and marked up the technical risk 
management model to indicate deficiencies or the model? 

2. What did you think of the model? 
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Appendix C: Letters Regarding "Ti n Whisker' 
FaUM f? MPti<i> 



United-gtatu Ornate 

comwnu 0 * 

%OVKMmWTai a//aim 


• luiUMuimi c* 

OvUfiAMT Of OOWUWUT WAM4(MtWT 
WAiMMOlO*, OC 10* II 


Stcaabar a, 1946 

The Honortbla Edward C. Aldridge 
Secretary 

ticpartaent of thi Air Forca 
The Pantaion 

Vaehlngton, D.C. 20330-1000 
Dear Hr. Secretary! 

Xc hu coat to ay attention that aoac alaotronlc part* 

In weapon* ayataat aay have failed, or be subject bo failure, 
due to th* growth of cryatala or «tln whlikera*. Thla 
phenoaenon appaara to raeult from the u*« of pure tin In 
oloae proilolty to alaotronlc parti. An aaaepla la tha uie of 
tin caaoa for houilng electronic hybrid*. Wnile several 
laontha aay bo required for tha tin whiskers to develop, they 
eventually grew to a length sufficient to bridge circuit*. 

X understand that at least two Major weapon systems require 
retrofitting sa a result of tho discovery of tin whicker*. 

Tht* raiaci acvaral questions X would Ilka answered by 
Dectcbcr 19» >946. 

Are tin caica ror electronic parti used in Air Force 
weapon oysters. If so, do these parts use currant of 
appro*lfltetfly 20 elcro isftrtl at approilastcly 10 volte? 

Have an: of these parte failed? Xf so, we» a failure 
analysts parforaad, including an analysis to dtltralnt 
whether tin whlakara were at fault? If so, what was the 
result of the failure analysla? 
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In addition to sltctronlci hound In tin cimi, ora 
thare othar Inatanoaa of failed alaetronic pirn attrlbutabla 
to tht oilatanoo of tin uhlaktrrt If so, plana dtaorlb# 
than lnatmeaa. 


• Thank you for your cooperation In thla cottar. If you 
have any question rejtrdlni thla request plena direct your 
staff to contact Brad Vena of ay ataff at 2C2-22M-36B2. 



CLiljt 





DEPARTMENT OP THE AIR FORCE 

HCtOOUAMTCM AghONAUTlCAl. SVSTCWS OVUiOm («# sC| 
MM)i*T^niMON AlA fOKI Ml, On*3 aval^sO) 

15 JAN 198 ? 


wwut Congressional Inquiry Regarding the Crowth of Crystals or a Tln Whiskers* In 
Electronic Parts (Sen Carl Levin's Ur, k Dee 86) 

HQ USAf/LEYY 

l. Tn« generic phenomena, growth of setsilie whiskers, was Identified 
during World War II with the growth of csdaluo whiskers on variable plate 
capacitors used in communication radios. Sines that tier, cadmium, stno, 
or tin whiskers have been found In crystal oscillators, capacitors, relays, 
connector*, tin-plated eopper conductors In printed wiring boards, and tin¬ 
plated part leads. Limited occurrences are docuaenied in the Government/ 
Industry Oats Eachange Program (0>DCf) data bank and the technical litera¬ 
ture. ml specs such as H1L-M-3B510 have addressed the latter issue for 
several years. These metal whiskers are undesirable as they have the 
potential to cause electrical disturbances within electronic components/ 
systems. 



2. Ue have also eiperlenced soee United esperlences with this Issue 
within S few components used In the F-15 radar and the ACH-65D 1* Maverick 
Missile. These occurrence# ease to light ee e result or ours and of our 
contractor's Internally generated aetlvltlss. The specifies ere outlined 
in paragraph J, for the F-15 radar; and In paragraph k, for the AGm-65. 

3. Gould Corporation was given s coniraet by ASD (the PRAM office) to 
deeanstrate the effectiveness of Combined Environmental Stress Screening 
(ESS) and Destructive Physical Analysis (DPI) prograe for ths surveillance 
of electronic parts returned from the field. The DPI identified tin 
whiskers In selected microchip packages. Subsequent inquires lndlcslsd 
that Hughes, El Scgundo CA, altered their original production process 
specification after one of their vendors had e Tire In the vendor's 
production facility. The F-15 radar uses a solder seal process for 
providing e hermetic seal for their hybrids using a pure tin plate on the 
hybrid lids to facilitate tne solder process. Originally these lids had a 
fused tin coaling (healed In eh oil oath to 2k0-26O°C to relieve stress 
within the material). After the change, the fusing process was deleted. 

The unfused tin lids have e high probability of growing tin wniskers while 
the fused lids ere stable. Tin whiskers have bevn found in a number of 
lids that have been resoved from fielded units or from spares Inventory. A 
small number of these unite have evidence which Indicates that tin whiskers 
have shorted between circuit conductors end part of the whisker has vaporlied. 
This could esuse sn Intermittent In the operation of the radar, in the 
event a tin whisker rests between two unprotected conductors within the 
hybrid, and If the hybrid at that location Is unable to deliver the current 
necessary to fuse the whisker, e hard failure could result. However, there 
is no evidence of whisker generated hard failures in F-15 radar hybrids. 
Hughes has modified their manufacturing processes lo preclude the growth In 

t in whiskers. 
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The ACM-65 IN K-veriek Missile hi* eaperlenced on* problem during factory 
acceptance testing of • IUt* of Acceleration Meter (NCi.1) provided by Systron- 
Ponner. This device eomsins a moving eoll mounted around a fliad permanent 
•ugnet which santts movements of tha IN glmbal plat fori and provides correc¬ 
tion signets to the global drlva motors. Failure analysis of lnoparatlva 
NOAH* found during missile assembly ahowad tin whisker* to b« growing froa 
a tin-plated solder pad on the aovabla coll which war* shorting the outer 
cast. This design Us been modified to raplaea th* lie-plated solder pad 
with one plated with btrylllua copper which has eliminated th* problem. 

Currant production incorporates this change. The epproalaately J00 alaalles 
del ivered to th* field are being retrofitted at the contractor*s eapens*. 

This retrofit is eapeotad to be ooapleied by the iumi: of 1167. There is 
no indication this problea has caused any failures in over NO launches by 
operational units. 

5- as a result of this experience th* following observations are sad•: 

a. W* are able to respond to these question* because our discipline* 
have been effective in identifying and correcting potictlal causa* of field 
failure in electronic equipment. 

b. U* havt realnded our program offices of the potential for Metallic 
whisker growth in electronic parts and the sppropriait control procedures. 

c. We have asked our Air Force-laboratories “t o~l Cent ify/d eve lop 
suitable, cost effective, alternatives to tin plating for th* solder 
sealing of hcractlc packages. 

a. We recommend that ths ALCs adopt an electronic piece part surveillance 
program utilising the OCA and CSS process, S* appropr*.tie. Proapt identifica¬ 
tion of th* causes Of field failures and t he feed back of this information 
to the appropriate equipment and part sanufaciurers will have a significant 
Upset on the operational capabilities of our weapon i/steas. 

6. We are pleased that our efforts were Instrumental in bringing this 
issue to a successful 


T (&JJ , 

rr.r.-:-- -; 7. r .'.i. a. 

TaiKVcs! I 'tcicr 
Doputy for E/.s‘.neeiir'3 


.us ion. 


V- 


1 Atch 

Sen Levin's ltr, 1 

cc: HQ Ar-:/5D 
HO Afit/FL 
AKWAl MLS 
ASD/Ck':PA) 
ASD/TtS 

Asa/i: 
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Appendix D: A Brief Chronology of the Development 
of the Rocket 


AD 1232 


1903 


1914 


1915 


1920-1922 


Nov 1923 


Late 1923 


Mar 1926 


Rocket-propelled arrows (somewhat like 
firecrackers) used in battle in China 
(145:26). 

Konstantin Eduardoviteh Tsiolkovsky's first 
article on rocketry appears in a Russian 
journal; Tsiolokovsky conceives of a rocket 
with a combination of a liquid oxygen and 
liquid hydrogen (145:41,42). 

Robert H. Goddard, American, granted patents 
covering "combustion chambers, propellent 
feed systems, and multistage rockets" 
(145:45). 

"Robert H. Goddard proved validity of rocket 
propulsion principles in a vacuum, at Clark 
University, Worcester, Mass" (42:4). 

"Goddard developed and unsuccessfully tested 
first liquid propellant engine, using liquid 
oxygen, and devised small high-pressure 
pumps to force fuel into the chamber" 
(42:16). 

"Robert H. Goddard successfully operated a 
liquid oxygen and gasoline rocket motor on a 
testing frame, both fuel components being 
supplied by pumps installed on the rocket" 
(42:17). 

"Die Rakete zu den Pla'netenaumen (The Rocket 
into Planetary Space) by Hermann Oberth was 
published in Germany, and was the genesis 
for considerable discussion of rocket 
propulsion" (42:18). 

"Robert H. Goddard launched the world's 
first liquid-fueled rocket at Auburn, 

Mass., which traveled 184 feet in 2 1/2 
(two-and-a-half] seconds. This event was 
the "Kitty Hawk" of rocketry" (42:21). 
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Nov 1929 

Dec 1930 

Apr 1932 

Mar 1935 

Mar 1936 

Oct 1936 


Dec 1941 

Jun 1942 

Oct 1942 


Famed aviator Colonel Charles A. Lindbergh 
visited Goddard. Through Lindbergh’s effort 
Goddard was awarded a $50,000 grant from the 
Guggenheim Fund and another grant from the 
Carnegie Institution (145:46). 

"Robert H. Goddard fired 11-foot liquid fuel 
rocket to a height of 2,000 feet and a speed 
of near 500 mph over Roswell, New Mexico" 
(42:26). 

"First flight of Goddard rocket with 
gyroscopically controlled vanes for 
automatically controlled flight" (42:29). 

"Goddard rocket attained altitude of 7,500 
feet in New Mexico" (42:33). 

"Robert H. Goddard's classic report on 
"Liquid Propellant Rocket Development" 
reviewing his liquid-fuel rocket research 
and flight testing since 1919, was published 
by the Smithsonian Institution" (42:34). 

"Lt. John Sessums (AAC) visited Robert H. 
Goddard to officially assess military value 
of Goddard's work. He reported that there 
was little military value, but that rockets 
would appear useful to drive turbines" 
(42:43). 

First test of the German V-l rocket at 
Peenmunade (145:105). 

"First test of t.ie German A-4 (V-2) rocket 
unsuccessful at Peene-mde, Germany" 

(42:43). 

■'First successful launch .d flight of 5 1/2 
[ f ive-and-a-half ] tor. German A-4 rocket (V- 
2) at Peenemunde, which traveled 120 miles" 
(42:44). 
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Appendix E: 

1905 

Dec 1938 

Jan 1939 

Aug 1939 

1939 

1940 

1941 

Dec 1941 
May 194 2 

1942 


Chronology of Some Major Events Regarding 
Atomic Bomb Development 


Albert Einstein publishes the Special Theory 
of Relativity that states the equivalence 
between energy and mass (46:829). 

Enrico Fermi receives Nobel Prize for 
experiments involving low velocity neutron 
bombardment of elements (45:324). 

Refugee Austrian scientists Lise Meitner and 
Otto Frisch explain (via Niels Bohr) that 
German scientists discovered "low-velocity 
neutrons cause the uranium nucleus to 
fission, [break apart]". . . much energy was 
released in the process" (46:324); numerous 
laboratory experiments begun (46:324). 

Albert Einstein's letter (dated 2 August 
1939) addressed to President Roosevelt, 
warns that the Germans may be developing the 
first atomic bomb, and that recent work by 
L. Szilard and Enrico Fermi has established 
technical feasibility (41). 

Alfred A. 0. Nier achieves the first 
separation of the uranium isotopes U-235 and 
U-238 at the Universi ,y of Minnesota 
(47:517). 

First funds for atomic research given to 
United States scientists (46:831). 

"The Columbia chain-reaction experiment with 
natural uranium and carbon yielded negative 
results" (45:325). 

The U.S. enters World War II (45:325). 

"The decision is made to proceed on all 
promising production methods [for obtaining 
fissionable materials)" (45:325). 

Army Corps of Engineers opens an office in 
New York City called the Manhattan Engineer 
District office (45:325). 
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Dec 1942 

Jun 1942 

July 1942 

Apr 1943 

1943 

Jul 1943 

Sep 1944 

Apr 1945 

Jul 1945 
6 Aug 1945 
9 Aug 1945 
16 Aug 1945 
Aug 1946 
Apr 1956 


"First nuclear chain reaction successfully 
accomplished at the University of Chicago" 
(42:44). 

J. Robert Oppenheimer appointed "director of 
Project Y, the group that was to design the 
actual weapon" (45:325). 

Experimental data reveals "plutonium does 
give off neutrons in fission, more than 
Uranium-235" (45:325). 

Seth Neddermeyer proposes an "implosion" 
method of attaining supercritical mass 
(45:325). 

John von Neumann and Edward Teller support 
the implosion approach (45:325). 

Summer/Fall "Gun" Method of attaining super 
critical mass of 1943 being pursued 
(45:325). 

"It [becomes) clear that the plutonium gun 
could not be built" (45:325), therefore, the 
implosion approach is pursued. 

"First reactor at Hanford, Washington, 
turned on but . . . promotly turned itself 
off" (45:325). 

Harry Truman briefed on the Manhattan 
Project, after just two weeks as president 
(45:325). 

"First test atomic device exploded in New 
Mexico" (42:50). 

An atomic bomb (using gun approach) is 
dropped on Hiroshima, Japan (42:51; 47:830). 

An atomic bomb (using implosion method) is 
dropped on Nagasaki, Japan (42:51; 47:830). 

"World War IT ended with Japanese surrender" 
(42:51). 

The Soviet Union tests their first atomic 
device (47:831). 

"Dr. John von Neumann was awarded the Enrico 
Fermi Award for anticipating the importance 
of the high speed computer in nuclear 
development programs. . . ." (42:82). 


171 







Appendix F: A Brief Chronology of the Development 
of the Laser 


1917 The principles of spontaneous and stimulated 
emission of radiation recognized by Albert 
Einstein (44:686; 101:23). 

1954 A working maser device is built by Townes, 
James P. Gordon and H. J. Zeiger using 
ammonia gas; subsequent improved models are 
built by Bell Laboratories. MASER is the 
acronym for Microwave Amplification by 
Stimulated Emission of Radiation (135:234). 


1958 

1959 

1960 

Jul 1960 

1961 

1961 

1961 

1963 


Arthur L. Schawlow and Charles H. Townes 
proposed a technique to obtain an optical 
MASER. Such a device would be based on 
Light Amplification by Stimulated Emission 
of Radiation (LASER) (134:234). 

A helium-neon gas laser is proposed by Ali 
Javan of Bell Telephone Laboratories 
(135:230). 

A successful prototype helium-neon gas laser 
built and demonstrated by Ali Javan, W. R. 
Bennett, Jr., and D. R. Harriott (135:230). 

The first successful construction and 
operation of a laser achieved by T. H. 

Maiman of the Hughes Aircraft Company using 
a synthetic ruby crystal (134:234; 135:227). 

The first gas laser is operated (Bell 
Laboratories) (135:266). 

Ivan P. Kaminow of Bell Laboratories 
demonstrates a high-frequency laser 
modulator that uses a potassium dihydrogen 
phosphate (KDP) crystal (117:333). 

Other lasers are constructed using: (1) 
different crystals, and (2) "a mixture of 
hydrogen and helium gas" (23:495). 

First liquid laser successfully operated by 
General Telephone and electronics (101:259). 
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Appendix G: A Brief Chronology for Achievement of Man, on 
the Moon 


Sept 1955 

4 Oct 1957 

31 Jan 1958 

2 Apr 1958 
12 Apr 1961 

5 May 1961 

25 Jan 1962 

1962 

1964 

1965 

9 Noy 1966 


President Eisenhower assigns "highest 
national priority" (42:79) to the research 
and development of the Intercontinental 
Ballistic Missile (ICBM). 

The Soviets launch Sputnik I—the world's 
first man-made satellite—into Earth orbit 
(11:32; 42:91; 145:164). 

United States launches Explorer I satellite 
into Earth orbit (11:32; 42:91; 145:164). 

Eisenhower creates the National Aeronautics 
and Space Agency (NASA) (11:32; 145:165). 

Soviet Cosmonaut Yuri Gagarin "orbited earth 
in a five-ton capsule becoming the first man 
in space" (1 32). 

United States President John F. Kennedy, in 
a speech to the Congress, announces the goal 
of a manned rocket mission the moon 
(116:736; 145:170). 

Congress approves a development program for 
the three-stage Saturn V launch vehicle; 
program given the highest priority 
(145: 4.70). 

U.S. Ranger 3 probe missed the moon 
entirely. Ranger 4 lands on moon but due to 
malfunction sends back no data. Ranger 5 
missus moon and loses power due to solar 
cell malfunction (145:191). 

U.S. Ranger 6 probe launched; unsuccessful 
due to burned-out electrical system. Ranger 
7 appropriately redesigned and successfully 
transmitted pictures of the moon (145:19). 

Soviet space probes Luna 5, 7, and 8 crash 
onto the moot.. Luna 6 misses the moon 
entirely (145:191). 

First launch of a Saturn V vehicle 
successful (90:60; 145:171). 
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3 Feb 1966 

30 May 1966 

Apr 1968 

27 Jan 1967 

Oct 1968 

Dec 1968 

Mar 1969 

May 1969 

Jul 1969 


Soviet space probe Luna 9 achieves first 
soft landing on the Moon's surface 
(145:191-192). 

U.S. Surveyor probe launched; makes 
successful soft landing on moon and 
transmits pictures (145:193). 

Second launch of a Saturn V vehicle—part of 
the unmanned Apollo 6—is unsuccessful due 
to failures of three J-2 engines on the 
vehicle (90:60; 145:171). 

Apollo 1 . Three astronauts killed by fire 
during full launch rehearsal (91:53). 

Apollo 7 . First manned flight of Apollo 
Command and Service vehicles conducted in 
Earth orbit; test fired Service Module's 
engine by firing it (while in orbit) 
automatically as well as manually; heat 
shield performance checked during reentry 
(91:53). 

Apollo 8 . Man's first flight around the 
moon; 147-hour mission. Twenty hours spent 
circling the moon. Photos of potential 
landing areas taken (91:53). 

Apollo 9. First manned flight with the 
Lunar Module; ten-day flight confined to 
earth orbit; separation, rendezvous and 
docki.ig practiced between the Lunar Module 
(LM) and the Command Module (CM) (91:53) . 

Apollo 10 . "Successful dress rehearsal for 
the actual moon landing (which occurred] two 
months later . . . first time complete 
Apollo spacecraft had orbited the moon" 
(91:53). This was an 8-day mission (91:53). 

Apollo 11 . First man on the moon; crew of 
Neil Armstrong, Edwin Aldrin, Mike Collins 
(91:54). 
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Appendix H: A Brief Chronology of the Development 
of the Jet Aircraft 


Nov 1917 

Dec 1919 

Mar 1922 

Jun 1920 

Sep 1928 

During 1929 

During 1931 

During 1932 

Apr 1937 


"Committee on Light Alloys established 
within NACA to develop new metals for 
aeronautical use, constructor Jerome C. 
Hunsaker was Navy member" (42:7). 

"The Aeronautical Engineering Society was 
organized at MIT" (42:11). 

"NASA report No. 159 on Jet Propulsion for 
Airplanes, by Edgar Buckingham of the Bureau 
of Standards, pointed out that jet fuel 
consumtion would be four times that of 
propeller engine at 250 mph, but that 
efficiency of jet increased at higher 
speeds" (42:14,15). 

"Initial flight of all-metal airplane 
(Gallaudet) designed by Engineering Division 
at Wright Field" (42:17). 

"Frank Whittle, RAF officer and engineer, 
obtained British patents for turbojet 
engine" (42:27). 

"NACA annual report indicated that 
aerodynamic efficiency may be increased by 
applying the principle of boundary layer 
control to the wings and possibly other 
parts of the airplane" (42:26). 

"Bureau of Standards made a number of 
experiments to determine whether the thrust 
reaction of a jet could be increased, and 
tested combinations of jets" (42:28). 

"German engineer, Paul Schmidt, working from 
design of Lorin tube, developed and patented 
a ramjet engine later modified and used in 
the B-l Flying Bomb" (42:29,30). 

"Frank Whittle's first gas turbine engine, 
the U-type, was static tested" (42:35). 
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1940 


Mar 1941 


Nov 1941 


Jul 1942 

Jul 1942 


Oct 1942 


Jul 1943 


Aug 1943 


"Committee of the National Academy of 
Sciences reported that operation of turbine 
wheels at temperatures up to 1,500 [degrees] 
F might soon be possible because of U.S. and 
foreign development of high-temperature 
alloys" (42:40). 

"NACA established Special Committee on Jet 
Propulsion to review early British reports 
on the Whittle engine, which subsequently 
aided development of TG-100 turboprop engine 
by GE and the 19-B turbojet by Westinghouse. 
Dr. W. F. Durand was called out of 
retirement to head this committee" (42:41). 

"Italian jet-propelled Caproni-Campini 
airplane flown 475 kilometers in 2 hours ll 
minutes from Turin to Rome, by Mario de 
Bernardini" (42:42). 

"German ME-262 turbojet fighter flown on 
spectacular flight test, concluding a series 
begun in May" (42:44). 

"First U.S.-designed jet engine successfully 
demonstrated at Langley Laboratory, the NACA 
Jeep, which was never flown but proved 
invaluable for continued NACA research on 
gas-turbine jet propulsion" (42:44). 

"First U.S. jet-propelled aircraft flight, 
by an Airacomet Bell XP-59A (powered by two 
1-16 engines developed by General Electric 
from the British Whittle prototype), made at 
Muroc Dry Lake, CA, with Robert Stanley as 
pilot" (42:44). 

"First turbojet engine completed for the 
Navy, the Westinghouse 19A, completed its 
100-hour endurance test" (42:45). 

"German turbojet fighter, a Messerschmitt 
ME-262, demonstrated before Adolph Hitler in 
East Prussia" (42:45). 
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Appendix I: Letter Regarding Electrostatic Discharge Damage 



DEPARTMENT OF THE AIR FORCE 
MHi&HT.ffcTTiMkOR aih roncc iaic Ohio m>i 


7 AUG 1985 


' Eh (PA) 

' Product Assuranee Policy, Electrostatic Discharge (ESD) 

*• ASC/AE ASD/AT ASD/B1 ASD/KW ASD/TA 

*50/YP ASO/TS A 50/YU ASD/YT ASD/YE 

(lor Assistant for Product Assurance) 

1 . Electrostatic discharge has become well recognised as s source of failure, 
delay and coat In electronics essvably ss well as of incipient failures affect¬ 
ing systea reliability and durability In operational service. Parts such as 
alcrocircults, discrete scalconductors, tnick snd thin flla resistors, hybrids 
and chips are known to be Sensitive to this phenoatnon. In conjunction with our 
general workmanship and quality program requirements, we have come to raped 
that our contractors dealing with such devices will have adopted effective ESD 
protection practices. 

2 . In a recent special survey conducted by APCMD. they not only found 
deficiencies In contractor ESD protection efforts, but also esperlenced a lack 
of contractual leverage on aoae prograas with which to secure contractor 
itprovcacni* in the £50 area. DOD-STO -1686 (1980). •Electrostatic Discharge 
Control Program for Protection of Elactronle Parts, Assemblies and Equipment,” 
is the preferred contractual vehicle for specifying the heed for ISO controls 
and should be cited In tnr product specification for all systcas containing CSO 
sensitive devices. 

3. In s review conducted for Cen McMullen, we found that DOD-STD-1686 la 
generally cited es a requirement on the newer ASD programs; but since product 
specifications on several current prograas predate the existence of this 
standard, it has not always been node contractually applicable. In such cases, 
we advise looking into the adequacy of your contractor 1 * E50 control practices 
In conjunction with the ATPAO to be sura that there are no alsund«ritano1ngs of 
our contractual Intent. It la our opinion tnat the M1L-Q-9858A quality pregr^n 
specification should suffice aa the governing requirement In auch Instances, 


v. Our near tern policy Is that DOD-5TD-1686 be referenced In all new product 

specifications for sysleas containing slat Lc-eenall Ive devices. This standard 

will be added to the listing of workmanship standards on the ASD Fora *$, 

Quality Assurance Memorandum. In the \o.:, berm, the Avionics Integrity Program 

nlL-Pftlht standard, now under development, will encompass the DOD-STO-I 660 
provisions and will be the primary means for contreclual1y Invoking ESD and 
otner integrity-related requirements In avionics ayateas acquisition. 


5. Should you have any questions about this policy or Its laplcaentatlon, 
please contact the undersigned or Mr George Thlelcn, ASD/tkSl, attention 


<3ohn c. halpim 


Assistant for Product Assurance 
Uputy for Engineering 


cc: ASD/EhSl 
Crus 
PnOO 
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a LOW-TECHNOLOGY 
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Appendix K: Interview with Dr. Keith Richev 
Technical Director of WRDC, 28 July 1989 


Interviewer : Sir, what are your technical qualifications? 

Dr. Richev : The highest degree is a PhD in Aerospace 
Engineering from the University of Michigan. 

Interviewer : What is your technical experience? 

Dr. Richev : I'll give you a biographical sketch, then 
I'll fill in some details. I have 28 years of experience 
in the government laboratory systems. In fact, all in 
Flight Dynamics laboratories of AFWAL now (reorganized 
into] WRDC. And for the last 2 years in WRDC front office. 
So, all of my experience except for 2 years I've gone to 
school full-time with a PhD level with the flight dynamics 
laboratory, and with WRDC. 

Interviewer : What is the mission of the organization of 

which you are a part? 

Dr. Richev : Basically research and development. 

Technology development for the Air Force. So, its a 
focused application of all technologies. Aerospace, 
electrical—any technologies that we deal with, focused 
toward Air Force applications. The basic research through 
advanced development. 

Interviewer : What is your present job? 

Dr. Richev : Technical Director of. WRDC. (Chief Scientist 
of the Chief Scientists, if you will). 

Interviewer : In your opinion, what are the most important 

techniques for managing technical risk? 

Dr. Richev : Well, first of all I think you have to 
understand the problem. And understand as much as you can 
about the technical characteristics of a problem. Try to 
be as analytical as you can. I guess the engineer in me is 
coming out. You try to be as analytical as you can but to 
recognize a lot of things that you can't be analytical 
about. But I think you need to sit back and try to 
understand the problem as much as you can. Then try to 
understand the various facets of the problem. I'll try to 
break it down into some elements. Try to understand how 
the elinteract. And then try to get a feel for the 
degree of risk--whether its achieving an objective, or 
schedule or cost, or anything else. A degree of risk that 
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might be associated with each element and the interaction 
among the key elements, and then try to construct a 
critical path function that shows you where the highest 
risk elements are and their importance to the outcome of 
the system. Something might be very high risk but not very 
important. So you wouldn't care about managing it at all. 
Let it be high risk. So what. It doesn't affect the 
outcome. Something else could be of lower risk but 
extremely critical. So, that a little bit of risk in a 
critical element could have a large affect on the outcome 
of the system. Try to identify those elements or most 
likely combination of elements because things don't usually 
happen singularly. They usually happen in multiples. 

Those combination of elements which are the highest risk to 
achieving the desired outcome. This assumes that you have 
a pretty good idea of what the outcome is, which is not a 
good assumption in all cases. But, let's say you have a 
good idea of the desired outcome; so having gone through 
this analysis it could be qualitative or quantitative. But 
you have some idea of the elements and the degrees of risk, 
and the importance. Now we have identified the elements or 
combination of elements that have the largest impact on the 
desired outcome. And as I said before some of those 
elements might be high risk factor; others might be the low 
risk factors. But the criteria is the effect on the 
outcome. So, having identified the elements of the problem 
that have the largest effect cn the outcome, now we would 
be in the position where we'd try to manage the risk of the 
elements. To me managing the risk means to identify what 
the risk is at the beginning of the process, then taking 
positive action to reduce the risk to an acceptable level 
at the conclusion of the process—recognizing that you 
can't wait until everything is at zero risk to complete. 
Most R and D (Research and Development] projects will be 
completed without having risk equal to zero in all factors. 
There is going to be some risk at the end. But it has been 
managed to be by the process. It has been managed to be 
reduced by the many factors to an acceptable level. Now a 
case in point in the Air Force labs WRDC particular, is 
reducing risk to an acceptable level so that the next stage 
of technology development can begin or follow on. Which 
would be engineer development. Category 6.4 budget, which 
is not done by the laboratory but is done by the system 
program offices. Whoever is applying the technology would 
take our product. Our product is technology developed to 
such a point that there is enough confidence in a low 
enough risk, but I wouldn't say zero risk. Enough 
confidence and low enough risk, so that they can apply the 
technology with confidence in the next step of development. 
Which is 6.4 engineering development.. And tnat is the way 
it is in the Air Force. That is the way it is in the 
evolution of the technology. Now my perception would be 
that in any given project there will still be risk, when we 
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hand it off from 6.3 to 6.4. The risk will be further 
reduced and managed in the 6.4 and beyond. But the 
laboratories wouldn't have much to do with that—with the 
risk management 6.4 and beyond, only on consultation 
basis. Because our job is to develop the technology to a 
point and then hand off. We very seldom stay with a 
technology all the way through to its final application. 
It's just the way the Air Force is organized. We have 
laboratories, SPOs, users. We have Logistics Command and 
so forth and, so on. If we took the attitude in the 
laboratories, that we have to reduce risk to an absolute 
zero level before we transition it, number one: we'll keep 
the technologies too long and they wouldn't transition to 
the Air Force as fast as they should. Two, it would be 
prohibitively expensive because the last 10 percent of a 
project, in terms of a full risk reduction, is very 
expensive. And you want to only do that in given 
applications. Because if we would say that we had to 
reduce the risk to zero for any and all applications since 
we don't know what the applications will be, we'll have to 
reduce it for any possible application, and that would take 
many more resources, in terms of dollars and people, than 
we have available. So, we make a conscious decision as to 
when to turn it loose. To use a reasoned judgment on what 
that hand off is. We sometimes learn after we hand it off 
that we didn't do enough. There is a major surprise 
somewhere in the application of our technology. And if 
were real honest with ourselves, we'd say we should have 
caught that. We should have known that would happen. The 
users of our technologies, in that case, give us a poor 
mark. The user comes back and says: you guys gave me 
foreign technology, and we say: well we probably did. We 
usually make an excuse that we didn't have enough money or 
didn't have enough time. And there is always the unknown 
unknowns. Once in awhile a technology goes sour in a big 
way after it leaves the laboratory. So we try to minimize 
that by lessons learned and by understanding the 
application of our technology to the field after they leave 
the laboratories. We try to learn as we go, and minimize 
our cases but, they can still happen. 

Interviewer : Based upon your experience, what is the most 

important technique for managing technical risk? 

Dr. Richey : It's hardly a technique, but I would say the 
answer to your question would be knowledge; as much 
knowledge as you can possibly have on the physical and non¬ 
physical phenomenon that you're dealing with. We don't 
have perfect Knowledge of any perimeters, but I would say 
that the single most important to know to manage risk is to 
understand the physical and nonphysical driver. 
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Interviewer : Does your organization have a specific 

written strategy or policy for assessing and managing 
technical risk? 

Dr. Richev : Not specifically. There are no documents that 
you can pick out—that I am aware of at least for the 
center level—that's called risk management or even the 
risk assessment. On the other hand it goes through 
everything we do. You know we manage risk every day. It's 
part and parcel to the management of technology. And 
knowledgeable people, scientist and engineers, and 
managers, and project managers, and technical development 
are—whether they recognize are managing risk every day. 

But to my knowledge there's not a written policy. Now 
there have been some studies done on risk assessment by the 
laboratories; risk assessment and perhaps risk management. 
And the most knowledgeable guy I would know of that has 
done this in WRDC is Dr. Squire Brown [Interviewed]. 
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Appendix L: Interview with Dr. Harris Burte 
Chief Scientist of WRDC Materials Lab. 

17 August 1989 


Dr. Burte : I understand better what you are driving at as 
we look at this outline [Preliminary Risk Management 
Model]—it's a rather thorough and complete outline of many 
things. I have two comments. One has to do with the idea 
that risk reduction means different things depending upon 
whether you're in the process of developing a system to 
meet a stated operational deficiency—at one extreme or at 
the other extreme exploring or doing what you can to foster 
the transition into use of new technological possibilities. 
Those are quite different things. In the middle between 
those [extremes] you might have the question of pursuing a 
national goal that's less well defined than a required 
operational deficiency that you have to pursue or pursuing 
the possibilities of meeting some national vision or goal 
that somebody such as the President enunciates. Examples 
of these might be "put a man on the moon" or the SDI 
project where the President or some other decision making 
authority has a vision of something and says "I want to get 
there." Those are three different levels. In one case you 
have a required operational need and [you want to solve] 
"What's the best way to meet it at this time ?" That's 
akin to the systems development process. In the other case 
you have a vision of something in the future and you are 
exploring possibilities. In the third case you're 
developing the capability to meet these, possibilities. In 
the final case, you have a technological possibility which 
might be applied to a wide variety of operational needs, or 
perhaps only a small number of operational needs and you 
are exploring what you have to do to transfer laboratory 
concepts or demonstrations into something that can actually 
be used. I have the feeling that you treat all of these a 
bit the same in this [model] and I think that you might 
want to explore the extent to which they are different. 

When we're working against a potential threat or a 
statement of operational deficiency, to use your words [in 
the model], the ways of doing it tend to fall into many of 
the things that you're talking about here [in the model]. 
That's what a SPO does when it tries to organize of 
analyze: What are the best ways of meeting this? What are 

the cheapest ways? What will give me the most bang for the 
buck? How do I cope with the risk? In these ways, balance 
the risks against the payoff and make sure that I'm not 
taking undue risks for the time and the budget I have to do 
my system. In the "mandated national priority" sort of 
thing, such as "put a man on put a man on the moon or do an 
SDI, there has to be some guidance as to timeline. At 
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different stages you can take a lot of risk, you can 
explore a lot of possibilities, or, in something as simple 
as SDI—"let's provide a defensive shield for the country" 
--risk isn't much applicable there in the initial stages. 
The question is to explore what the possibilities are more 
at that stage. And then when you have the possibilities, 
to assess the question of what it takes to get them into 
use. Risk becomes a factor as a function of time and 
budget available to put it into use. In the exploration 
stage, risk is considered differently. Risk is: Which 
things are most likely to be promising in the long range? 
That's different than: How much time and money will it 
take me to get all the bugs out of an actual operational 
system? Those are quite different questions. Finally, in 
the case of taking something where some degree of 
laboratory feasibility has been demonstrated—I talked 
about this to some extent last time [earlier interview]— 
and transitioning it. The question is now: you've got a 
laboratory thing, but if somebody is building a system, he 
has a certain time and budget and he's got to meet that 
time and budget so he has to know that the things he 
chooses for his system are compatible with his kind of 
budget. An industrial analogue to that is: if somebody 
wants to bring a new product to the market, he does a 
return on investment calculation. But to do the return on 
investment calculation in the commercial world often 
requires a great deal of information about the new product, 
and he requires the knowledge that he will be able to do 
these things [that] strange things won't pop up and hit him 
in the face and destroy him. 

Therefore, the decision to go ahead with a new 
product—the capitalizing, build the factory, do all the 
advertising and everything to get a new product to the 
market—is r. big multi-buck decision. If there is a lot of 
risk in terms of the new product—we're not really going to 
be able to do it or strange things are going to happen, or 
it's going to take us a lot more money or a lot more time 
to bring it to the market place—then the decision takers 
are likely not to go with the new product. We may go with 
it foolishly as Rolls Royce did [mentioned earlier 
interview] and go bankrupt first. So, those are different 
elements of risk reduction. In some cases it is a case of 
risk assessment. In the case of a system developer, it's 
more a case of risk assessment. You don't have that much 
time to reduce risk once you decide that you want to go to 
a system. In the case of pushing new technology forward, 
then there is the question of learning what the strange 
things are or: How can I scale it up? Or making sure 
there are no gaps in it or learning more about the behavior 
of the new product so that you can really assess what the 
new benefits will be versus the cost of it. In the case of 
exploring new possibilities, like in the SDI example, 
that's a totally different question. There one has to 
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[ask], which possibilities are most likely to give me the 
biggest payoff? Now at a much later stage, somebody might 
decide now I want to set up an SDI system. Of the 
different possibilities available, in which one is the risk 
adequate for my purpose? That becomes a risk assessment 
issue. 

Interviewer : So, risk assessment doesn't start until you 

completely or comprehensively explore the possibilities? 

Dr. Burte : Well, you can always do risk assessment at any 
time. People in the military in particular—you're 
spending money to take chances. But at any given time you 
may say I have got to use this technology [or] I won't be 
able to do what I want to do. But you can take an 
assessment of what the risk is and ask yourself then.: Are 
you willing to do that? You shouldn't just go ahead without 
thinking about it. So those are the main points as i see 
it. And I think those are a bit different and I think you 
have a bit of all of those confused. In other words for 
the three things of mandated major possibility like SDI or 
the Orient Express (National Aero-Space Plane] at the 
present—that's for the question of the early stage. For 
each of them [the three sources of risk shown in the model] 
the question of risk is considered differently. If it is a 
requirement, I want to do this in a finite time—like put a 
man on the moon. That can be like a system requirement and 
you can say, well, for that time and budget I have, is the 
risk tolerable or can I overcome the thing? If it is a 
mandated national thing that says: I will build an Orient 
Express that's a different thing. There we're exploring 
the possibilities—or SDI as initially put forth. And if 
in fact you're restricting yourself to where the President 
says I want a definitive capability at such a time at such 
a budget, then it is a risk assessment issue. But there 
may be time in that to do the development of new 
technology. Much of what I was talking about wasn't in the 
risk assessment aspect it had to . with the question of 
converting new technology to a poj. where somebody could 
do a credible risk assessment. If it's too early, you're 
making too many assumptions about the risk; you don't have 
the knowledge of what the risks are. Part of risk 
reduction is just getting the information so that you can 
make a risk assessment. Okay, those are the main things 
that I want to say and that may or may not apply very much 
to this (risk management model]. If you're sticking very 
much to [the case where] the technology is essentially 
there to do the job or to do something like the job that I 
want, within the time and budget that I have, trying to 
pick the technologies that will best work for the purpose, 
what you've written tends to apply considerably. If you're 
getting into other things like an SDI system, or finding 
uses for new technology such as VHSIC—that's much of what 
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I was talking about beforehand [in the earlier interview] 
and you may not be getting into that. You may want to look 
over what you're saying from the viewpoint of it [the 
model] creeping into that. Because that's risk reduction. 
Here you're talking about risk assessment and assuring that 
the risk in the first case where you've got a definitive 
set of requirements and now you want to make sure that the 
approaches that you take fit that. That's not really risk 
reduction. That's risk assessment and risk selection— 
unless you have some time to explore different 
possibilities. If you have time to explore different 
possibilities, that becomes something like risk reduction. 
But many of these [projects] don't give you much time to 
explore different possibilities. . . . Risk management 

gets to be not so much risk reduction as risk assessment— 
choosing the right things—when you're in the systems 
world. You should have a lot of that done before you go 
into the system selection. Essentially, risk management is 
different than concept exploration. 

The other thing I want to talk about is what is the 
role of a government lab in this. There I feel strongly 
that a government lab such as our lab or any other 
government lab—one of their highest callings is to play a 
major leadership role in doing any of the things I have 
talked about. A government lab hopefully is wedded only to 
the Air Force--in the case of an Air Force Lob. It's a 
neutral ground; it has no ax to grind. Hopefully it has 
within itself excellence which is the peer of anybody in 
the country, not just bureaucratic smart buyers. Really 
competent people who know the technology and are on the 
forefront of leading the development of new technology. 
People such as this should be deeply involved in any of the 
things I've talked about; risk assessment for giving a 
clear-minded unbiased viewpoint of what we know and what we 
don't know. And an assessment of how serious is what we 
don't know and how long it will take us to fill in the gaps 
and is that commensurate with the budqet and time available 
for the project that you've set up. In the case of 
exploring a spectrum of new possibilities, the excellence 
involved here is that which can exercise leadership in the 
nation as a whole. To have very good people explore new 
possibilities in other than a wish-is-the-father-to-the 
conclusion mode. Exploring them enthusiastically but 
always asking: What do we know and what don't we know and 
what might the pitfalls be? The government lab can again 
exercise tremendous leadership here and act as a governor 
on that process because the people who are advocates for a 
given activity tend to enthusiastically look at the 
potential for success. As [the late Nobel prize winning 
physicist Richard P. ] Feynman said—I think his words were 
something like in the Challenger [Space Shuttle disaster 
review] committee—I think his words were something like 
"you can't fool Mother Nature." That's the point. People 
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who really understand the science and the technology and, 
by being wedded to the Air Force, make sure we ask the 
right questions. The other point which is the business of 
assessing and understanding what it takes to reduce the 
risk. That's all the work that takes place as you go 
between the typical 6.2 work and the introduction of 
something into a system. It is that work—when I talked to 
you last—that I was saying is deficient in many areas of 
the country. 
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Interviewer ; Sirs, what are your technical qualifications? 

Dr. Olsen : I hold a bachelors, masters, and PhD all in 
Aerospace Engineering. 

Dr. Haver : I have a bachelors in Mechanical Engineering, a 
masters in Engineering Mechanics, and a PhD in Mechanical 
Engineering. 

Interviewer : What is your technical experience? 

Dr ._O lsen : I've done,aircraft performance, instability and 

control and then went into structures, vibrations, flutter 
unsteady aerodynamics. 

Dr. Maver : I was a mechanical engineer with Bell Telephone 
laboratory supporting the mechanical and thermal 
development of electronic devices, electron tubes, and 
integrated circuitry, and also principal engineer. I was 
also Cryogenic Engineer with Grumman Aerospace, also I was 
the principal engineer with Cornell Aeronautical labs, SCAD 
Missile project, with the Air Force worked on R&D into 
aircraft environmental control systems, Electronics 
coolant, a little bit of windshield impact research. I 
have to get into a lot of diversified things because of the 
position I hold as Chief Engineer of Mechanical Division 
where we cover landing gear, escape systems and 
environmental control, and ballistics research. 

Interviewer : What is the mission of the organization of 

which you are a part? 

Dr. Olsen ; Our mission is to do the research and the 
development that gives the Air Force the capability to 
develop a better airplane and to improve the ones they have 
or to give them a range of options of what they might want 
to choose for the next aircraft. We do a research and 
development and aero dynamics, structure, flight control, 
system and all the subsystems. 

Interviewer : What are each of your present jobs? 

Dr. Olsen : I'm the Chief Scientist of the [Flight 
Dynamics] the laboratory. 
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Dr. Maver : For the ihicle Subsystem Division, I'm the 
division's Chief Assistant for Research and Technology. 

Interviewer : In each of your opinions, what are the most 

important techniques for managing technical risk? 

Dr. Olsen : Risk is something I don't think we manage that 
much. We generally working on trying to prove sorething is 
feasible or something can be done. How about assessment? 

I guess explicitly I don’t think we practically speaking 
really evaluate the risk. We talk about risk. But we 
generally determine if the idea is possible. In a very 
general way if it's practical At least I don't know of 
any way really quantify risk in deciding to go with higher 
risk or lower risk. 

Dr. Maver : We compare various alternative concepts and do 
payoff analyst. We try to establish the feasibility and 
the process of establishing feasibility. I think we 
discover that some things are not feasible therefore too 
risky and other things are more feasible. And those that 
survrve these various levels of investigation, they 
eventual!'/ get developed. And our contractors evaluated 
alternative approaches. They propose approaches and if we 
think they're credible by virtue of having checked the 
concepts versus the physical laws of nature then we just 
keep going as long as it remains feasible. If we run 
against an obstacle. Because we've in the business of 
eliminating risk or sorting out risky things from 
impossible things, just by continuing to develop them. If 
we hit a stone wall then the project stops, or some other 
alternative attack is pursued. 

Dr. Olsen : I guess if you're developing an airplane you 
want to look at several different approaches of doing a 
particular part of the mission or building part of the 
airplane. You look at the risk there. But I guess in my 
experience in the 61XX and 62XX areas really risk is not 
something you evaluate very much Maybe in 63XX development 
there might be some efforts they pursue or don't pursue 
because of risks. 

Interviewer : What is the most important technique for 
managing technical risk? 

Dr. Olsen : I'll go back and say that I don't think that we 
really have some kind of graduated scale of risk that we 
manage against. We try to put our resources where they pay 
off the most and we try to do things that we can prove are 
feasible and will have high payoff. 

Interviewer : How do you improve that feasibility? 
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Dr. Olsen : Well, I guess you come up with the possible 
design or application or implementation you can have. But, 
again the feasibility is sort of either it is or it isn't. 
It's like, it works or it doesn't work. For instance we 
developed something called a active control system to 
aircraft flutter the vibrations of the wings. And we 
demonstrated to our satisfaction that the 61XX and 62XX 
level that we could make it work. We went into the wind 
pile and built it. Yes, this thing works. And we know how 
to do it. When the question about risk comes that sort of 
moves up into another ball game. That question is: can we 
get the F-16 program office to use this method to improve 
their airplane? At this point they're saying either it is 
to expensive, or it doesn't come at the right time or is a 
too high of a risk. But again its a"go" or "no go" 
decision. So I don't know what the best method is of 
managing risk. 

Interviewer : Does your organization have a specific 

written strategy or policy for assessing and managing 
technical risk? 

Dr. Olsen : Hot that I'm aware of. We have one set of 
programs that we pursue called the laboratory independent 
programs where the director tells us that he wants us to go 
in to very risky areas. Because of the perception that 
sometimes we get too conservative. So there is a whole pot 
of funds set aside to go into those high risk areas to see 
what you can do. 

Dr. Maver : We make use of analysis to see if our concept 
is feasible. Using mathematical, physical models of things 
to the extent we can anticipate that. Another step is to 
actually do development testing to do tests to determine 
some phenomenon performs. Or we can get the performance out 
of the phenomenon that we look for in magnitude. And we 
eventually try to develop a demonstration. And there is a 
lot of iterative steps between analysis and testing until 
it's clear the right combination of the attributes and 
performance characteristics have been achieved. We also 
worry about cost, weight. Is the thing to heavy? is it 
palatable? It can't weigh to much, it can't cost to much, 
and it has to be energy efficient. So, it is a balancing 
act, and sometime we switch concept in order to alleviate 
some of the objectionable characteristics. 

Dr. Olsen : I think that you're going to find risk in the 
laboratories is not the same as risk in the SPO. He’s got 
a plane to build and a time to build it in, and a cost to 
build it in. He's definitely looking for the lowest risk 
way of getting done what he has to get done. 
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Dr. Haver : We though provide him with the information on 
which concepts are feasible, because they have been tried 
in the laboratory previously. I think a SPO director would 
be foolish not to require that something have been 
demonstrated previously in an advanced development program 
where we spend a lot of money just trying to do something 
for the first time to prove that it can be done. And its 
done in a quasi generic fashion so, that when the SPOs want 
to build a particular aircraft, they derive some confidence 
measure from our having done a generic version of that 
previously. 

Dr. Olsen : We demonstrate whether it works at all, or 
whether if it can be done at all. 

Interviewer : Can I have a copy of any strategy for 

minimizing technical risk? 

Dr. Olsen : I don't know of any. 

Dr. Maver ; Well, I suppose that you could consider the way 
we categorize the levels of research as basic research 
where natural phenomenon are investigated basically to 
understand, the exploratory research, where we try to apply 
these phenomena and configure them to work and perform some 
useful function on an aircraft or Air Force contracts. 

Then there is advance development, where some form of 
realistic full scale demonstrations of the subsystem or 
product it nducted. And then there is the 64XX, the 
engineering .evelopment, for a particular weapon system. 
That is what ASD and the SPOs is exposed to. 

These also correspond to budget categories. The 
"appropriate money set aside are kind of inversely with the 
risk. The further stages really get more expensive, 
because you have to pay attention to more realistic 
details. You have to make it look like an airplane, make 
it look like a wing, a landing gear, or a tire. Whereas at 
the early stages, we can just play with the materials, 
rubber or what ever. 

Dr. Olsen : I think that is a good point, because I guess 
when you move up to basic research, exploratory. Are you 
familiar with the 61XX, 62XX, 63XX? The 61XX is basic 
research, 62XX is exploratory development. And as you move 
into the other categories like 63XX and 64XX then risk. . . 

In the 63XX program they do look for different options 
to achieving the goal and evaluating the risk. [Is 6lxx, 

62XX, 63XX, part of the labs?] Yes, even some 64XX. 

Dr. Maver : We actually build some windshields through 
retrofits. But we primarily use 61XX, 62XX, 63XX. 
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Dr. Olsen : The programs are 61XX for basic, 62XX for 
exploratory, 63XX for advanced, and 64XX is for engineering 
development. I guess there is otxier [budget categories] 
categories 65XX but I don't know what they are. Maybe you 
should talk to some 63XX people. I was trying to think of 
the guys who does the 63XX programs for structures. 

There's a guy you should talk to named Vern Johnson. 
Johnson's phone number is X55664. And I know that they 
look at part of their 63XX program, and they look at 
several different approaches to get a particular 
improvement in an airplane. And one of the things they 
evaluate is the risk. Risk versus cost performance. They 
try to avoid risk because they're now in the exploratory 
and basic. We're more "blue sky" and willing to be more 
daring, and try different things. As it gets closer to 
something that goes on an airplane people begin to manage 
the risk as you say by eliminating risky things and 
pursuing less risky design options. 

Dr. Olsen : I think we just didn't really get into it [risk 
reduction] until we start talking about the 63XX that 
Arnold [Dr. Mayer] brought up. 

Dr. Maver : That's where the aircraft or aerospace aircraft 
manufacturers get into it. They are usually the performers 
of those contracts. And they force them to do things the 
way they already know how to do them. To some extent that 
minimizes risk too. 

Dr. Olsen : I don't know if you would want to minimize the 
risk. I think its just an all out effort. Maybe later on 
when you got to the point of where, you've got a bomb built 
and you're going to put it on an airplane to drop it, then 
minimizing risk might become an issue. 

Dr. Maver : Well, they had to work out a lot of problems. 
How do you make sufficient quantities of these radioactive 
materials for the bomb. And I guess they have to develop 
some of the factories. Some of the uranium purifications 
centrifuge and whatever is involved. 

Dr. Olsen : And then there were a couple of different 
process. I remember that General Groves was forcing the 
people to pursue at least two different manufacturing 
methods. The manufacturing methods and minimizing risks. 
And giving himself at least an option. 

Dr. Mav er: Now we use that too, we call it fly offs. We 
usually try to have two different teams go and build 
something. Then they fly off the best result so you'll at 
least have the tetter of two approaches. 
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Dr. Olsen : Of course the engineers wanted to keep the 
parallel path going as long as possible (for the atom 
oomb]. Eventually General Grove forced them into a 
decision to go one way or the other. Whether the way they 
do the enclose or how they manufacture the stuff. That's a 
good history to look at. I always thought that he was just 
the civil engineer that built the things and put them 
together. The last book 1 read, he really played a very 
strong management role. He wasn't just someone that was 
standing on the side lines waiting on for results, but he 
was really driving those guys. 

Dr. Maver : Well, I don't know whether putting the man on 
the moon is one of your cases (it is]. I recall that one 
way they minimize risk is that they had, one engineer on 
every piece of hardware. There was one engineer that did 
nothing but one particular valve for three years. And he 
was 100 percent responsible for that valve. The design and 
performance. I guess if you put a lot of people on 
something and dedicate them you can reduce risk. 
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Appendix N: Interview with Dr. Jess Riles. Chief Scientist 
o f the Avionics Laboratory. Wright Research and 
Development Center. 25 July 1989 

Interviewer : Sir, what are your technical qualificatio;.. 

Dr. Rile s: I hold a bachelor in Mechanical Engineering, 
bachelors in Aeronautical Engineer?ng, and a masters and 
PhD in Electrical Engineering. 

Interviewer : What is your technical experience? 

Dr. Riles : At one time I was the program manager for NASP 
(National Aero-Space Plane) National back in the 60s' when 
it was called the Aero-Space Plane, now it is the National 
Aero-Space Plane. I have worked in research and 
development. I have worked in the acquisition business in 
Aeronautical Systems Division. When I was in the service I 
was a Maintenance and Communication Officer. So I kind of 
had the full spectrum of creating the idea to how you keep 
it working on the flight line. 

Interviewer : What is the mission of the organization of 

which you are a part? 

Dr. Piles : Research and development, and Aerospace 
Avionics. 


What is your present job? 

Dr. Riles : Chief Scientist of the Avionics Laboratory, 
Wright Research and Development Center. 

Interviewer : In your opinion, what are the most important 

techniques for managing technical risk? 

Dr. Riles : So you're saying that the first elements of 
feasibility have been assessed? 

Interviewer : Yes. 

Dr. Riles : Well, I think probably the most used technique 
here at ASD (Aeronautical System Division) is to put people 
on it with systems engineering experience, that have 
developed similar kind of things in the past. To try as 
much as possible to support at least a couple of competing 
approaches. These days with the funding we have its a 
little difficult to fund competing approaches. So 
sometimes you prematurely have to decide on, if you will, 
"one horse to ride." And after you have gone a year or two 
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downstream, you find you're on the high risk horse. If you 
could, you could fund the beginning approaches. The other 
is to solicit the opinion of many people in the other 
services and other national agencies, to help you in 
establishing their course of engineering development of 
whatever you're talking about, and we do a lot of that. We 
bring in expert co,. ...ittees, scientific advisory boards. We 
bring in expert consultants from industry and from other 
sources other national labs to help us. 

Interviewer : Based on your experience, what is the most 

important technique for managing technical risk? 

Dr. Riles : I think that probably the one that seems to 
have worked most often is make sure you use people that 
have experience. What's, the old saying? "When you don't 
know what direction you're going, almost any direction will 
do" . 


Interviewer : Does your organization have a specific 

written strategy or policy for assessing and managing 
technical risk? 


Dr. Riles : Well, see the way you justified the risk, 
you've almost taken the problem out of the laboratory. 
Because what we do is come up with some new ideas and we 
carry them through to some kind of feasibility 
demonstration. Then we try to encourage users. "Users" is 
a rather broad term to us it can be people in the operating 
commands. It could be people in the intelligence agencies, 
it could be the engineering people in ASD who support SPO's 
in going out and acquiring weapon systems. So, once you 
take it out of that "demonstrate the feasibility" it's a 
kind with technical viability of a concept for use of the 
Air Force or other services. Its then something that's 
outside the laboratory to manage to get into a weapon 
system. 


Interviewer : Let's say that you have a lot of feasible 

options, how does the lab decide which one has the least 
technical risk? 


Dr. Riles : Well, you're kind of getting into the 
philosophy of how do we prioritize the work we do. What 
you're talking about is an investment strategy. And what 
we do is that in the various disciplines that we're 
responsible for like electronic warfare, offensive 
avionics, developing things to go out and find targets, 
identify them and provide the weapons capability to strike 
them. We build programs. First of all, we provide some 
top down guidance, and how much money we have, what does 
front office think we should be in what kind of businesses, 
which ones look like we should invest in the heavier, and 
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then from the bottom they build the program. They put 
together the various ideal that we want to pursue. Now 
some of them are way out. Some of them are closer in. 

Then we have what we call kind of a rack. It's a period of 
time, usually the early part of the given year where we 
develop the next year program. And during that time we 
have to evaluate the maturity of the concept, and how much 
risk there is involved in doing this and is it absolutely 
frivolous to go out and do something or is there some 
possibility it's going to pay off. Usually we take expert 
opinion inside, in doing our first ranking of these ideal. 
If we have questions about the liability of some 
particular piece of technology, we have experts that we can 
go to: scientific advisory boards, defense science boards. 
We all have people in the industry we can talk to or 
technical compatriots in universities that we can go access 
and get their opinions to help us decide whether or not we 
have ranked or prioritized or agree to fund this idea in 
the appropriate fashion. Or is there to much ra.sk in it? 
It's just not worth investing in now. So I would have to 
say that expert opinion is probably the most used. 

Interviewer : Is there any policy within the organization 

for managing technical risk? 

Dr. Riles : A policy document? 

Interviewer : Yes. 

Dr. Riles : No. 


Dr. Riles : One thing that I didn't mention is that we have 
a voting process, which involves risk in it and its 
essentially a Delphi method of choosing things to fund. We 
typically take a vote after we make sure that everybody 
understands the particular things that we want to fund. 

And by the way you talked about it. We have about 700 
different things going on at once, and about 100 new items 
are added each year and about a 100 die. So we typically 
have a steady state of about 700 programs, about 100 new 
ones starting, and 100 old ones completing. So, what we do 
is rank the continuing ones with the new ideas, and we use 
Delphi to do that. And a part of Delphi involves 
consideration and risk. We will fund things that have high 
risk if it has high pay off. 


202 






Appendix O: Interview with Dr. Squire Brown. Chief 
of the Technology Assessment Division. Wright 
Research Development Center. 28 July 1989 


Interviewer : Sir, what are your technical qualifications? 

Dr. Brown : I have a bachelor of science and a PhD in 
Aerospace Engineering, some 22 years of professional 
experience mostly related to aircraft technology, aircraft 
design, development of aircraft related technologies. 

Interviewer : What is your technical experience? 

Dr. Brown : I primarily worked in aerodynamics and flight 
vehicle design and analysis, from aerodynamics research to 
configuration development, analysis of operational 
effectiveness. 

Interviewer : What is the mission of the organization of 

which you are a part? 

Dr. Brown : The Technology Assessment responsibility is to 
examine and create systems concepts for future mission 
requirements for the Air Force, and to examine the 
technologies that are essential for the development or 
employment of that system. What are the technologies that 
are needed to be effective in the future scenarios? 

Interviewer : What is your present job? 

Dr. Brown : I'm Chief of the Technology Assessment Division 
[Wright Research and Development Center (WRDC)]. 

Interviewer : In your opinion, what are the most important 

techniques for managing technical risk? 

Dr. Brown : Deal a little bit with the term risk. Can we? 
Would you like to help me out with what you mean by risk? 

Interviewer : Assuming that technical feasibility has been 

established, how do we go from the establishment of 
feasibility to minimizing the risk of actually achieving 
that project. I guess a contemporary example would be the 
Space Defense Initiative or the contemplated X-ray laser. . 


Dr. Brown : the accepted approach, and I believe one that 
works well, is one that would lead through a series of 
hardware demonstration programs. It might be peculiar to 
the technologies, but it might be a combination of 
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component bench testing, simulation flight demonstration, 
either on a test bed or perhaps as part of a new prototype 
system. An X-vehicle [Experimental vehicle] or some 
suitable modification of a flight vehicle. But ultimately 
risk in the sense that we're talking about here almost need 
verification of the article. That would go into a deployed 
system. 

Interviewe r: Based on your experience, what is the most 

important technique for managing risk? 

Dr. Brown : It's a loaded question. Let me try to break it 
apart a little bit. We approach the demonstration in terms 
of getting the risk out as an accepted way of doing 
business. We recognize that technologies come along and 
show promise in the feasibility demonstrations, and we know 
that they're going to be critical to some capacity in the 
future system. And you've got to simply understand that 
you got to go ahead and do the flight demonstration part, 
or a suitable demonstration. Much of our energy is 
consumed , I wouldn't say so much as with management as 
with the advocacy and developing the programs to carry out 
this demonstration. The management would only occur in so 
far as our management of programs themselves. We are very 
much captive of the institutional way that we do business 
of the POM [Program Objective Memorandum] cycle. Ara you 
familiar with the 61XX, 62XX, 63XX programs? [Yes]. 

As we come out of 61XX is a basic research, much of 
the in-house work; much of the feasibility work is the 62XX 
arena limited amount of dollars. As we come out of 62XX 
saying this is the technology that we really need to flight 
demonstrate as for an example. Much of our energy is 
devoted to getting the dollars to get the program put in 
place to do that flight demonstration. A very excellent 
example at the moment is the 2-D (two-dimensional] 
vectoring nozzles that are flying on the F-15. Would you 
like me to go through that example for? [I think that I 
would prefer you to go through it.] 

It is probably as good an example as we have at the 
moment. It's relevant to your questions about how we 
manage risk. Because the idea several years ago looking 
into advanced systems, or advanced concepts. We thought 
that airplanes like the ATF should have a 2-D vectoring 
nozzle. Having decided that, then we had to look at what 
was the real capability to produce the flying article 
within weight, specs, and if it would be reliable. The 
chief engineer would incorporate that in the system. A lot 
of features: improved take-off performance—up and away 
maneuverability. In some cases considerations are embedded 
in that. Generally, a lot of added capability. There's 
clearly some risk embedded in that and part of that risk 
had to be with cost and weight issues. The management of 
that risk took the form of saying ground test alone is not 
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adequate. And subsequently, that has proven out. In fact, 
Pratt and Whitney did boiler plate—almost feasibility 
study—and test cell, put in a new engine and now you can 
see the thing move and all of that. But as they tried to 
transition that into a flight weight article they 
encountered a lot of difficulties in manufacturing and 
performance. Now, we eventually got there; the thing is 
flying. But it is still not [without problems] but it will 
establish a database for production people on the ATF 
[Advanced Tactical Fighter Aircraft] to use. Even the ones 
that are flying right now are not suitable for putting on a 
production airplane. We're going to have to go one stage 
more. So our management of the risk took the form of 
saying, first in the paper studies: this will help to 
expand the flight of the envelope of the airplane. The 
ground test itself was recognized through .recognize through 
professional judgement. The ground test itself is not 
adequate. When we built some flight articles, that those 
were still not suitable because of the problems we 
encounter. They were still not suitable as a production 
article. So there will be one more stage to go through 
this for the ATF program. All they do is the production at 
the very end of that. There is still some risk to that 
program office, that they may not meet all their goals in 
terms of cost, or weight or features in the long run. But 
our risk management took the form of assessing at each 
stage what we thought we'd do, what was positive, 
efficient, what was good to know, what critical issues did 
we have to address at the next stage. That's largely 
embedded for 2-D nozzles or anything. It's largely embedded 
in the programmatic of the cycle of POM submittals and 
annual budget reviews of program advocacies. Now generally 
its not called risk. Risk is a term that is not used very 
much. It is usually described more in traditional things 
like: does this thing weight too much, or does it function 

as well as it should, or took too long to manufacture. 

That is kind of our way of saying, well at the stage of 
that we have risk embedded there because we didn't [meet 
our goals], but I'm quite sure we can meet our goals. The 
lab rarely describes things in terms of risk as the 
management schools teach this king of favorite subject of 
mine. But it's not something that is a conscious part of 
the way we do business. 

Interviewer . Does you organization have specific written 
strategy or policy for assessing and managing technical 
risk? 

Dr. Brown : First of all, the mission statements recognize 
the need to go through the sequence that I just described. 
In order to eliminate risk. They don't necessarily look at 
a program. They very seldom look at risk typically. I'll 
show you this chart, and say for instance, this is a very 
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systematic way we do studies. And we start with a kind of 
a classical process for each program. We come out at the 
bottom at the end having come out of a technology 
assessment. And talk about technology goals and notice 
that risk is in there. If you look at our formal mission 
statements and the way we tell people we do business we 
hope to achieve what's in the risk. Survivability 
constitutes the one-on-one engagement process. If I'm 
looking at a penetration system, first I get the concept of 
it, then I look at the concept of one-on-one situation. 

And get the piece of this. Then I go into a more 
aggregated effectiveness analysis in which I may look at 
the total mission problem. And perhaps the number of 
airplanes dealing with the number of threat sites, or the 
number of supporting systems. Then begin to understand 
what the total mission problem. .And perhaps the number of 
airplanes dealing with the number of threat sites, or the 
number of supporting systems. Then begin to understand 
what the total effectiveness is in an aggregated sense. 
Survivability may mean, as it flies along at a particular 
speed at an altitude at a particular flight path, maybe 
stable, maybe maneuvering and has a certain signature, runs 
across a certain encounter. What is the probability 
survival there? One airplane against one threat site. 

Then I may have the total mission of reaching the target 
and dropping the bomb or whatever. Maybe it is composed of 
a number of sequential elements that I have at the end. 

And the next step, I'll have to kind of total all of those 
up. And try to decide if the system in aggregate was 
reasonably effective. 

We has a request some years ago by Military Airlift 
Command to look at what might be done for C-130 
replacement. Our designers started sketching out what they 
know about the mission. We begin to look at how the 
airplane would be used to begin to refine the designs. We 
looked at some one-on-ones: engagements with triple A 
sites. Also, we looked at how we might vary the 
performance of the airplane. Did it need to go faster? 

Have a lower signature? and begin to establish those 
tradeoffs. And we begin to compare this airplane with 
perhaps a more traditional variant of the transport. All 
through this there are clearly technologies that are 
required to make the airplane feasible. Once we satisfy 
ourselves that this airplane is a good candidate versus 
that. Suppose we decided that this is really better than 
that one way or the other. We'd then say the reason that 
this was successful is that it had the advance materials, 
and this advance gear box, and this advanced crew station. 
In order to survive an engagement like this, crew is going 
to have to have the following information and be able to 
act on it expeditiously. And that may require a few 
displays of evaluating those in a simulator or ultimately 
perhaps a flight article. If you can go back and look at 
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technology assessment, if you went to a functional division 
for example, if you went to a mechanics division and talk 
to them about technology assessment they have figures of 
merit that are uniquely associated with what they do with 
their science. You might need a life to drag [ratio of] 
and airfoil. And if tney can improve a lift to drag of the 
airfoil, they've had a success. We try to look at 
technology in terms of the whole system and it may be that 
the new airfoil does make some contribution. It may turn 
out, in fact, that the real key to success is a redesigned 
inlet--that is the most important thing that can be done. 
The total system picture we would come back out with the 
recommendation with the proqram and inlets. Then the guys 
down in XR [ASD Office of Plans] and other places may take 
a look at the whole scenario of the war, and say, what we 
really need is airplanes that will fly faster or something. 
They don't really care if this kind of thing needs a better 
inlet or something. So, that is kind of how we put 
technology into perspective. 
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Appendix P: Interview with Mr. Jack Cannon. Technical 
Director of Development Planning. ASP Office 
of Plans. ASD/XR. 25 July 1989 


Interviewer : Sir, what are your technical qualifications? 

Specifically, what technical degrees do you hold? 

Mr. Cannon : I simply hold a bachelor in science with a 
major in physics with considerable graduate work above the 
level. 

Interviewer : What is your technical experience? What 
technical jobs have you held? 

Mr. Cannon : I worked as a physicist in the materials 
laboratory for about five years and the rest of my career 
has been at the staff levels: a director of research of 
Wright Air Development Center, and later at ASD. I've been 
in systems planning since about 1963. 

I nterviewer : What is your present job? 

Mr. Cannon : My position here is Technical D'rector of 
Development Planning [A3D/XR]. 

Interviewer : What is the mission of the organization of 
which you are a part? 

Mr. Cannon : The mission of the organization basically is 
to essentially develop new system starts—and when 1 say 
new system starts it's not only totally new systems but 
major modification to existing systems—to solve major 
command deficiencies. Virtually all of the systems you 
will find today came through this kind of a process— 
through XR [the Office of Development Planning]. Things 
like the F-15, the A-10, the C-5, the B-l, the ATF 
[Advanced Tactical Fighter] the NASP [National Aero-Space 
Plane]. All of those programs started in XR and we take 
them up to the point of having done the concept exploration 
studies and then transition them to an acquisition agency, 
to a SPO [Systems Program Office] or to a deputy for 
acquisition. 

•Interviewer : In your opinion, what are the most important 

techniques for managing technical risk? The is assuming 
that [a system] has been demonstrated feasible in the 
laboratory. 

Mr. Cannon : You're saying breadboard feasibility? 
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Interviewer : Yes. So from that point what is necessary to 

manage the technical risk? 

Mr. Cannon : Well, generally, my feeling would be that you 
would have to go into an advanced development program which 
we generally speak of as being a, so that it's prototype 
hardware to demonstrate the concept. It could be 
demo. Jtrated on a ground test level in some cases to 
satisfy your concerns about risk or it might have to be 
flight demonstrated. But generally, you would not develop 
flight qualified hardware, you would build prototype 
hardware that would be flown in a prototype system before 
you would go the full nine yards of fully developing the 
system to prove out the concept. You would prototype it 
using as much existing technology that you can to 
demonstrate it before we get into new technology at the 
same time. 

Interviewer : Based on your experience, what is the most 

important technique for managing technical risk? 

Mr. Cannon : Well, you know if you had to select one I 
think that certainly the prototype testing is the way you 
reduce the risk. 

Interviewer : Does your organization have a specific 
written strategy or policy for assessing and managing 
technical risk? 

Mr. Cannon : Well, what is used in ASD is first of all, is 
whenever a 63XX [budget designation] program is started, it 
goes through what is called the SENTAR process. The deputy 
of engineering of ASD is really responsible for the SENTAR 
process. What they do is review at the start of the 63XX- 
program. I'm talking about a advanced development 
hardware demonstration program. They will review that 
proposed advanced development plan, and particularly 
concentrate on setting some criteria that they felt should 
be met chrough this demonstration. Some performance that 
would be demonstrated in the hardware demonstration. Then 
once the laboratories and the engineerii * people have 
signed up to that, and say that this is /hat we're going to 
do, and this is what the lab plans to do, and this is what 
is going to be necessary to do in order to satisfy the 
engineers that the technology is mature and is documented 
in a transition plan. The old twist was basically that at 
the same time you involve a customer. The customer could 
be an existing SPO or an existing system organization, if 
there is no existing system organization it then becomes XR 
as the customer for that technology. You will have a three 
prong agreement: a customer agreeing that it is useful 
technology in terms of the future, a laboratory which is 
developing the technology, and an engineering organization 
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which is geared to look over the results of the laboratory 
program to make sure that it has satisfied the 
predetermined criteria. And therefore, the technology 
would be considered mature at that time. You will probably 
want to look into the SENTAR process, which stands for 
Senior Engineering Technology Assessment Review. It is a 
group of tech directors out of the deputy for engineering, 
that form the SENTAR group, and they review all laboratory 
63XX-programs. The 63XX-program stands for advanced 
development. The programs that are used in the Air Force, 

61XX program is basic research, 62XX program is exploratory 
development, and 63XX program is advanced development. The 
63XX program is broken down into A&B. 63XXA is advanced 
technology, and 63XXB is systems oriented development. If 
you we're going to build a flying prototype, for example 
prototype it would be built under 63XXB. Electronic 
components would be built under 63A in that sort of broad 
technology area. Then the 64XX program is engineering 
development. Ideally something would flow from basic 
research to exploratory development to advanced development 
to engineering development. I say ideally because you'd 
probably find that many things don't always go in that nice 
classical fashion. 

Interviewer : Anything else you vould like to say about 

technical risk? 

Mr. cannon : Well, when we do advanced studies here in XR, 
I'm talking about things that are going to be operational 
in [the year] 2010. Our first look-see at technical risk— 
we would be in our laboratories and ask for technical risk 
analysis. In other words, did the lab see technologies 
that would make this system possible in the year 2010? 

We've been able to describe the system in terms of its 
performance: for example, its speed, range, altitude, 

maneuver and so forth. Then we ask the laboratories 
basically to give us a technology assessment in things 
like, materials, structures, avionics, propulsion which 
make up the system, and they give us an initial technical 
risk assessment. And then they begin to show that they 
have technology that they feel is going to satisfy that 
requirement or they will show that no they don't have and 
here is what will have to be done in the years you got as 
lead time for the system to begin to develop some 
technology. So I think that the key to us in this business 
is that we get that initial tech assessment done by the 
laboratory folks, who understand what technologies they 
really have in hand and what they really have to do yet in 
order to satisfy a given system requirement. As they 
imitate things, then the SENTAR process begins to be 
involved in terms of following, tracking what is happening 
on that advanced development type effort. 
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Interviewer ; Thank you a lot for your time, 
it. 


I appreciate 
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